從HDFStore檢索多索引熊貓數據框時出現問題（表格格式）

Question

我有一些代碼，簡化爲下面的示例，它包含一些原始數據，從中創建一個數據透視表，然後將它與另一個數據幀合併，最後將結果存儲在HDFStore對象中。如果以固定格式存儲，則可以很容易地找到它。但是，如果以表格格式存儲，則會產生錯誤。我需要表格格式，以便我可以一次提取塊（總數據集爲數千萬行）。

任何想法是什麼問題？

代碼示例：

import pandas as pd 

def createFrame(): 
    data = { 
      'colA':[1,1,1,2,2,2,3,3,3,4,4,4,4], 
      'colB':[5,5,5,5,5,5,5,5,5,5,5,5,5], 
      'colC':['a','b','c','a','b','c','a','b','c','d','e','f','g'], 
      'colD':['ap','ap','ap','bp','bp','bp','cp','cp','cp','dp','dp','dp','dp'] 
      } 
    frame = pd.DataFrame(data) 
    return frame 

def createOtherFrame(): 
    data = { 
      'colD':['ap','bp','cp','dp'], 
      'colE':[100,200,300,400] 
      } 
    frame = pd.DataFrame(data) 
    return frame 

if __name__ == '__main__': 
    pd.set_option('display.width', 120) # default is 80 
    pd.set_option('io.hdf.default_format','table')  

    pivotStore = pd.HDFStore('test.h5') 
    frame = createFrame() 
    otherFrame = createOtherFrame() 
    pivoted = frame.pivot_table(['colB'], 
           index=['colA'], 
           columns='colC', 
           aggfunc='sum' 
           ) 
    print(pivoted) 
    print('')  
# print(pivoted.info(),end='\n\n') 

    mergedFrameA = pd.merge(frame[['colA','colD']].drop_duplicates(), 
         otherFrame, 
         how = 'left', 
         on='colD' 
         ).set_index('colA') 
# print(mergedFrameA.info()) 
    print(mergedFrameA) 

    mergedFrameB = pd.merge(pivoted,mergedFrameA,how='left',left_index=True,right_index=True) 
# print(mergedFrameB.info()) 
    print(mergedFrameB) 

    pivotStore['bob'] = mergedFrameB 
    pivotStore.close() 
    store = pd.HDFStore('test.h5') 
    extracted = store.select('bob',start=0,stop=5) 
    print(extracted) 
    store.close() 

輸出產生（有誤差）：

 colB       
colC a b c d e f g 
colA        
1  5 5 5 NaN NaN NaN NaN 
2  5 5 5 NaN NaN NaN NaN 
3  5 5 5 NaN NaN NaN NaN 
4  NaN NaN NaN 5 5 5 5 

    colD colE 
colA   
1  ap 100 
2  bp 200 
3  cp 300 
4  dp 400 
     (colB, a) (colB, b) (colB, c) (colB, d) (colB, e) (colB, f) (colB, g) colD colE 
colA                       
1    5   5   5  NaN  NaN  NaN  NaN ap 100 
2    5   5   5  NaN  NaN  NaN  NaN bp 200 
3    5   5   5  NaN  NaN  NaN  NaN cp 300 
4   NaN  NaN  NaN   5   5   5   5 dp 400 
Traceback (most recent call last): 
    File "C:\multiindextest.py", line 52, in <module> 
    extracted = store.select('bob',start=0,stop=5) 
    File "C:\Anaconda3\envs\py34\lib\site-packages\pandas\io\pytables.py", line 665, in select 
    return it.get_result() 
    File "C:\Anaconda3\envs\py34\lib\site-packages\pandas\io\pytables.py", line 1359, in get_result 
    results = self.func(self.start, self.stop, where) 
    File "C:\Anaconda3\envs\py34\lib\site-packages\pandas\io\pytables.py", line 658, in func 
    columns=columns, **kwargs) 
    File "C:\Anaconda3\envs\py34\lib\site-packages\pandas\io\pytables.py", line 3999, in read 
    cols.set_names(names, inplace=True) 
    File "C:\Anaconda3\envs\py34\lib\site-packages\pandas\core\index.py", line 529, in set_names 
    idx._set_names(names, level=level) 
    File "C:\Anaconda3\envs\py34\lib\site-packages\pandas\core\index.py", line 3274, in _set_names 
    'Length of names must match number of levels in MultiIndex.') 
ValueError: Length of names must match number of levels in MultiIndex. 
Closing remaining open files:test.h5...done 

Answer 1

不能存儲這樣的指標，即部分元組和部分字符串。它不是一個真正的MultiIndex（也沒有任何用處）。

我只是簡單地使用常規索引。你也可以用MultiIndex做到這一點，但每一列都必須成爲它的一部分。

In [67]: pivoted = frame.pivot_table('colB',index='colA',columns='colC',aggfunc='sum') 

In [68]: pivoted 
Out[68]: 
colC a b c d e f g 
colA        
1  5 5 5 NaN NaN NaN NaN 
2  5 5 5 NaN NaN NaN NaN 
3  5 5 5 NaN NaN NaN NaN 
4 NaN NaN NaN 5 5 5 5 

In [69]: df = pd.concat([pivoted,mergedFrameA],axis=1) 

In [70]: df 
Out[70]: 
colC a b c d e f g colD colE 
colA          
1  5 5 5 NaN NaN NaN NaN ap 100 
2  5 5 5 NaN NaN NaN NaN bp 200 
3  5 5 5 NaN NaN NaN NaN cp 300 
4 NaN NaN NaN 5 5 5 5 dp 400