處理非常大的數據幀

Question

現在我在處理數據並將其轉換爲數據框時遇到了問題。基本上我試圖做的是閱讀第一

data = pd.read_csv(querylog, sep=" ", header=None) 

的數據，那麼這組

query_group = data.groupby('Query') 
ip_group = data.groupby('IP') 

，並最後創建一個空白的數據幀映射他們的價值觀

df = pd.DataFrame(columns=query_group.groups, index=range(0, len(ip_group.groups))) 

index = 0 
for name, group in ip_group: 
    df.set_value(index, 'IP', name) 
    index += 1 
df = df.set_index('IP') 

for index, row in data.iterrows(): 
    df.set_value(row['IP'], row['Query'], 1) 
    print(index) 
df = df.fillna(0) 

所以我問題是ip_group可能會上升到6000和的大小query_group最多400000這將導致在製作我的內存無法處理的非常大的空白數據框。任何人都可以幫助我解決這個問題嗎？任何幫助表示讚賞。數據

樣品數據幀是這樣

data = pd.DataFrame({ "Query" : ["google.com", "youtube.com", "facebook.com"], 
    "IP" : ["192.168.0.104", "192.168.0.103","192.168.0.104"] }) 

和我的預期輸出應該是這樣的

   google.com youtube.com facebook.com 
IP    
192.168.0.104 1   0    1 
192.168.0.103 0   1    0 

Answer 1

IIUC可以使用get_dummies，但沒有數據是有問題找到最好的解決辦法：

df = pd.get_dummies(data.set_index('IP')['Query']) 
print df.groupby(df.index).sum() 

樣品：

import pandas as pd 

data = pd.DataFrame({ "Query" : ["a", "b", "c", "d", "a" , "b"], 
    "IP" : [1,5,4,8,3,4] }) 
print data 
    IP Query 
0 1  a 
1 5  b 
2 4  c 
3 8  d 
4 3  a 
5 4  b 

#set index from column data 
data = data.set_index('IP') 

#get dummies from column Query 
df = pd.get_dummies(data['Query']) 
print df 
    a b c d 
IP    
1 1 0 0 0 
5 0 1 0 0 
4 0 0 1 0 
8 0 0 0 1 
3 1 0 0 0 
4 0 1 0 0 

#groupby by index and sum columns 
print df.groupby(df.index).sum() 
    a b c d 
IP    
1 1 0 0 0 
3 1 0 0 0 
4 0 1 1 0 
5 0 1 0 0 
8 0 0 0 1 

儘量轉換成int8通過astype節省3倍的內存：

print pd.get_dummies(data['Query']).info() 
<class 'pandas.core.frame.DataFrame'> 
Index: 3 entries, 192.168.0.104 to 192.168.0.104 
Data columns (total 3 columns): 
facebook.com 3 non-null float64 
google.com  3 non-null float64 
youtube.com  3 non-null float64 
dtypes: float64(3) 
memory usage: 96.0+ bytes 

print pd.get_dummies(data['Query']).astype(np.int8).info() 
<class 'pandas.core.frame.DataFrame'> 
Index: 3 entries, 192.168.0.104 to 192.168.0.104 
Data columns (total 3 columns): 
facebook.com 3 non-null int8 
google.com  3 non-null int8 
youtube.com  3 non-null int8 
dtypes: int8(3) 
memory usage: 33.0+ bytes 

print pd.get_dummies(data['Query'], sparse=True).info() 
<class 'pandas.sparse.frame.SparseDataFrame'> 
Index: 3 entries, 192.168.0.104 to 192.168.0.104 
Data columns (total 3 columns): 
facebook.com 3 non-null float64 
google.com  3 non-null float64 
youtube.com  3 non-null float64