熊貓read_table多列定義

Question

我有一個代碼生成文本數據，其中診斷輸出附加到單個文本文件在運行過程中。根據我的設置，將會進行不同的測量，並在每次運行開始時使用相關的標題行。輸出類似於此：

# time diagnostic_1, diagnostic_2 
0.3 0.25376334 0.07494259 
1.7 0.3407481 0.03018158 
2.2 0.45349798 0.85539953 
3.4 0.22368132 0.52276335 
4.8 0.17906047 0.40659944 
# time diagnostic_1, diagnostic_3 
3.4 0.65968555 0.67085918 
4.8 0.2122165 0.80855038 
5.1 0.96943873 0.41903639 
6.8 0.16242912 0.91949807 
7.0 0.68513815 0.22881037 
8.8 0.83304083 0.02394251 
9.2 0.01699944 0.58386401 
# time diagnostic_2, diagnostic_3 
8 0.79595325 0.8913367 
9 0.46277533 0.47859048 
10 0.30773957 0.64765873 
11 0.19077614 0.39109832 
12 0.0020474 0.44365015 

有沒有辦法有閱讀指定的字符串，而不是指定的行數後後pandas.read_table回報？周圍的工作我現在是做第一遍用grep來找到在哪裏劈叉是，使用numpy.loadtxt

該吐出與我想要的信息的數據幀

from subprocess import check_output 
import numpy as np 
import pandas as pd 
from itertools import cycle 

fname = 'foo' 
headerrows = [int(s.split(b':')[0]) 
       for s in check_output(['grep', '-on', '^#', fname]).split()] 
# -1 to the range, because the header row is read separately 
limiters = [range(a, b-1) for a, b in zip(headerrows[:-1], headerrows[1:])] 
limiters += [cycle([True, ]), ] 

nameses = [['t', 'diagnostic_1', 'diagnostic_2'], 
      ['t', 'diagnostic_1', 'diagnostic_3'], 
      ['t', 'diagnostic_2', 'diagnostic_3']] 
dat = [] 
with open(fname, 'r') as fobj: 
    for names, limit in zip(nameses, limiters): 
     line = fobj.readline() 
     dat.append(pd.DataFrame(np.loadtxt((s for i, s in zip(limit, fobj))), 
           columns=names)) 

Answer 1

完整的腳本加載陣列。有更新和刪除列的猴子業務是必要的，以保持複合指數。 retval.merge(dset, how='outer')給出相同的列，但是是一個整數索引。

from subprocess import check_output 
import numpy as np 
import pandas as pd 
from itertools import cycle 

fname = 'foo' 
headerrows = [int(s.split(b':')[0]) 
       for s in check_output(['grep', '-on', '^#', fname]).split()] 
# subtract one because header column is read separately 
limiters = [range(a, b-1) for a, b in zip(headerrows[:-1], headerrows[1:])] 
limiters += [cycle([True, ]), ] 

nameses = [['t', 'diagnostic_1', 'diagnostic_2'], 
      ['t', 'diagnostic_1', 'diagnostic_3'], 
      ['t', 'diagnostic_2', 'diagnostic_3']] 

with open(fname, 'r') as fobj: 
    for names, limit in zip(nameses, limiters): 
     line = fobj.readline() 
     dset = pd.DataFrame(np.loadtxt((line for i, line in zip(limit, fobj))), 
          columns=names) 
     dset.set_index('t', inplace=True) 
     # if the return value already exists, merge in the new dataset 
     try: 
      retval = retval.merge(dset, how='outer', 
            left_index=True, right_index=True, 
            suffixes=('', '_')) 
      for col in (c for c in retval.columns if not c.endswith('_')): 
       upd = ''.join((col, '_')) 
       try: 
        retval[col].update(retval[upd]) 
        retval.drop(upd, axis=1, inplace=True) 
       except KeyError: 
        pass 
     except NameError: 
      retval = dset 
print(retval)