MySQL：用FROM子句中的相關子查詢重寫MSSQL？

Question

我們有一個包含網站的頁面訪問量，像一個表：

time  | page_id 
----------|----------------------------- 
1256645862| pageA 
1256645889| pageB 
1256647199| pageA 
1256647198| pageA 
1256647300| pageB 
1257863235| pageA 
1257863236| pageC 

在我們的生產表，目前大約40K行。我們要生成，每天的獨特網頁在過去30天裏瀏覽，60天，90天計數。因此，在結果集中，我們可以查找了一天，看到獨特頁是多少那一天之前的60天期限內訪問。

我們能夠得到一個查詢的MSSQL工作：

SELECT DISTINCT 
CONVERT(VARCHAR,P.NDATE,101) AS 'DATE', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE NDATE BETWEEN DATEADD(D,-29,P.NDATE) AND P.NDATE) AS SUB) AS '30D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE NDATE BETWEEN DATEADD(D,-59,P.NDATE) AND P.NDATE) AS SUB) AS '60D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE NDATE BETWEEN DATEADD(D,-89,P.NDATE) AND P.NDATE) AS SUB) AS '90D' 
FROM PERFLOG P 
ORDER BY 'DATE' 

注：由於MSSQL不具備FROM_UNIXTIME功能，我們增加了測試NDATE列，它僅僅是轉換time。生產表中不存在NDATE。

這個查詢轉換到MySQL爲我們提供了「未知科拉姆P.time」錯誤：

SELECT DISTINCT 
FROM_UNIXTIME(P.time,'%Y-%m-%d') AS 'DATE', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 30 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS SUB) AS '30D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 60 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS SUB) AS '60D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM (SELECT PAGE_ID FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 90 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS SUB) AS '90D' 
FROM PERFLOG P 
ORDER BY 'DATE' 

我明白這是因爲我們不能有一個相關子查詢，在外部FROM子句引用的表。但是，不幸的是，我們在如何將這個查詢轉換爲在MySQL中工作時遇到了困難。現在，我們只需返回表中的所有DISTINCT行，並在PHP中進行後處理。 40K行花費大約2-3秒。當我們有100個1000行的行時，我擔心這個表現。

是否有可能在MySQL中嗎？如果是這樣，我們可以期望它比我們的PHP後處理解決方案表現更好。

UPDATE： 這裏的查詢創建表：

CREATE TABLE `perflog` (
    `user_id` VARBINARY(40) NOT NULL , 
    `elapsed` float UNSIGNED NOT NULL , 
    `page_id` VARCHAR(255) NOT NULL , 
    `time` INT(10) UNSIGNED NOT NULL , 
    `ip` VARBINARY(40) NOT NULL , 
    `agent` VARCHAR(255) NOT NULL , 
    PRIMARY KEY ( `user_id` , `page_id` , `time` , `ip`, `agent`) 
) ENGINE MyISAM 

我司生產的表有40K〜行迄今！

Answer 1

爲什麼你還要埋在第二級這樣子查詢？試試這個：

SELECT DISTINCT 
FROM_UNIXTIME(P.time,'%Y-%m-%d') AS 'DATE', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 30 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS '30D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 60 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS '60D', 
(SELECT COUNT(DISTINCT SUB.PAGE_ID) FROM perflog WHERE FROM_UNIXTIME(time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 90 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d')) AS '90D' 
FROM PERFLOG P 
ORDER BY 'DATE' 

Answer 2

你可以嘗試使用單選。

選擇日期90天前僅之間的值。

然後使用的情況下，語句中的每個fiels以檢查日期30，60，90之間落入對於每個字段，如果情況屬實，那麼1否則爲0，並且計數的那些。

喜歡的東西

SELECT SUM(CASE WHEN p.Date IN 30 PERIOD THEN 1 ELSE 0 END) Cnt30, 
     SUM(CASE WHEN p.Date IN 60 PERIOD THEN 1 ELSE 0 END) Cnt60, 
     SUM(CASE WHEN p.Date IN 90 PERIOD THEN 1 ELSE 0 END) Cnt90 
FROM Table 
WHERE p.Date IN 90 PERIOD 

Answer 3

更改子查詢到連接，因爲這樣：

select 
    FROM_UNIXTIME(P.time,'%Y-%m-%d') AS 'DATE', 
    count(distinct p30.page_id) AS '30D', 
    count(distinct p60.page_id) AS '60D', 
    count(distinct p90.page_id) AS '90D' 
from 
    perflog p 
    join perflog p30 on FROM_UNIXTIME(p30.time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 30 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d') 
    join perflog p60 on FROM_UNIXTIME(p60.time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 60 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d') 
    join perflog p90 on FROM_UNIXTIME(p90.time,'%Y-%m-%d') BETWEEN DATE_SUB(FROM_UNIXTIME(P.time,'%Y-%m-%d'), INTERVAL 90 DAY) AND FROM_UNIXTIME(P.time,'%Y-%m-%d') 

然而，這可能緩慢，因爲堆的殺戮你的日期列任何indicies功能的運行，更好的解決方案可能是：

create temporary table perf_tmp as 
select 
    FROM_UNIXTIME(P.time,'%Y-%m-%d') AS 'VIEWDATE', 
    page_id 
from 
    perflog; 

create index perf_dt on perf_tmp (VIEWDATE); 

select 
    VIEWDATE, 
    count(distinct p30.page_id) AS '30D', 
    count(distinct p60.page_id) AS '60D', 
    count(distinct p90.page_id) AS '90D' 
from 
    perf_tmp p 
    join perf_tmp p30 on p30.VIEWDATE BETWEEN DATE_SUB(P.VIEWDATE, INTERVAL 30 DAY) AND p.VIEWDATE 
    join perf_tmp p60 on p60.VIEWDATE BETWEEN DATE_SUB(P.VIEWDATE, INTERVAL 60 DAY) AND p.VIEWDATE 
    join perf_tmp p90 on p90.VIEWDATE BETWEEN DATE_SUB(P.VIEWDATE, INTERVAL 90 DAY) AND p.VIEWDATE; 

Answer 4

這是我用來解決這個問題的PHP。理想情況下，我希望這一切都由MySQL完成（如果可以更快地完成）。我只發佈任務本作進一步澄清：

function getUniqueUsage($field = 'page_id', $since = 90){ 
    //we need to add 90 days onto our date range for the 90-day sum 
    $sinceSeconds = mktime(0, 0, 0, $m , $d, $y) - (($sinceDays + 90) * (60 * 60 * 24)); 
    //==> omitting mySQL connection details<== 
    $sql = "SELECT DISTINCT From_unixtime(time,'%Y-%m-%d') AS date, $field FROM perflog WHERE time > $sinceSeconds ORDER BY date" ; 
    $sql_results = mysql_query($sql); 
    $results = array(); 
    //all page ids per date (ending-up with only unique date keys) 
    while ($row = mysql_fetch_assoc($sql_results)) 
    { 
     $results[$row['date']][] = $row[$field]; 
    } 
    $sums = array(); 
    //initialize sum array, with only unique dates (days) 
    foreach (array_keys($results) as $date){ 
     $sums[$date] = array(0,0,0); 
    } 
    //calculate the 30/60/90 day unique pages for each day 
    foreach (array_keys($sums) as $ref_date){ 
     $merges30 = array(); 
     $merges60 = array(); 
     $merges90 = array(); 
     $ref_time = strtotime($ref_date); 
     $ref_minus_30 = strtotime("-30 Days",$ref_time); 
     $ref_minus_60 = strtotime("-60 Days",$ref_time); 
     $ref_minus_90 = strtotime("-90 Days",$ref_time); 
     foreach ($results as $result_date => $pages){ 
      $compare_time = strtotime($result_date); 
      if ($compare_time >= $ref_minus_30 && $compare_time <= $ref_time){ 
       $merges30 = array_merge($merges30, $pages); 
      } 
      if ($compare_time >= $ref_minus_60 && $compare_time <= $ref_time){ 
       $merges60 = array_merge($merges60, $pages); 
      } 
      if ($compare_time >= $ref_minus_90 && $compare_time <= $ref_time){ 
       $merges90 = array_merge($merges90, $pages); 
      } 
     } 
     $sums[$ref_date] = array(count(array_unique($merges30)),count(array_unique($merges60)),count(array_unique($merges90))); 
    } 
    //truncate to only specified number of days 
    return array_slice($sums,-$since, $since, true); 
} 

正如你可以看到，有很多不幸的陣列合併，ING和陣列獨特-ING的。

Answer 5

注：我在閱讀@astander，@ Donnie，@longneck的解決方案後寫這篇文章。

我知道性能很重要，但爲什麼不存儲聚合？十年的每日行是3650行，每列只有幾列。

TABLE dimDate (DateKey int (PK), Year int, Day int, DayOfWeek varchar(10), DayInEpoch....) 
TABLE AggVisits (DateKey int (PK,FK), Today int, Last30 int, Last60 int, Last90 int) 

這樣，您只需在一天結束時僅運行一次查詢，僅一天。預先計算的總量是任何高性能分析解決方案（多維數據集）的根源。

UPDATE：
你可以通過引入另一列DayInEpoch int（天數自說1990-01-01）加快這些查詢。然後您可以刪除所有這些日期/時間轉換功能。