如何更快地遍歷這個文本文件？

Question

我有很多部分在這個格式的文件：

section_name_1 <attribute_1:value> <attribute_2:value> ... <attribute_n:value> { 
    field_1 finish_num:start_num some_text ; 
    field_2 finish_num:start_num some_text ; 
    ... 
    field_n finish_num:start_num some_text; 
}; 

section_name_2 ... 
... and so on 

該文件可以是幾十萬排長隊的。每個部分的屬性和字段數可以不同。我想建立一些字典來保存這些值。我已經有一個單獨的字典，它包含所有可能的'屬性'值。

import os, re 
from collections import defaultdict 

def mapFile(myFile, attributeMap_d): 
     valueMap_d = {} 
     fieldMap_d = defaultdict(dict) 

     for attributeName in attributeMap_d: 
      valueMap_d[attributeName] = {} 

     count = 0 
     with open(myFile, "rb") as fh: 
      for line in fh: 
       # only look for lines with < 
       if '<' in line: 
        # match all attribute:value pairs inside <> brackets 
        attributeAllMatch = re.findall(r'<(\S+):(\S+)>', line) 
        attributeAllMatchLen = len(attributeAllMatch) 
        count = 0 

        sectionNameMatch = re.match(r'(\S+)\s+<', line) 

        # store each section name and its associated attribute and value into dict 
        for attributeName in attributeMap_d: 
         for element in attributeAllMatch: 
          if element[0] == attributeName: 
           valueMap_d[attributeName][sectionNameMatch.group(1).rstrip()] = element[1].rstrip() 
           count += 1 
         # stop searching if all attributes in section already matched 
         if count == attributeAllMatchLen: break 

        nextLine = next(fh) 

        #in between each squiggly bracket, store all the field names and start/stop_nums into dict 
        #this while loop is very slow... 
        while not "};" in nextLine: 
         fieldMatch = re.search(r'(\S+)\s+(\d+):(\d+)', nextLine) 
         if fieldMatch: 
          fieldMap_d[sectionNameMatch.group(1)][fieldMatch.group(1)] = [fieldMatch.group(2), fieldMatch.group(3)] 
         nextLine = next(fh) 

     return valueMap_d 

我的問題是，while循環，所有的字段值相匹配是明顯比其餘代碼的要慢0.5秒與2.2S根據CPROFILE如果我刪除了while循環。我想知道我能做些什麼來加速它。

Answer 1

當你需要花哨的模式匹配時，正則表達式非常棒，但是當你不需要時，使用str方法解析文本會更快。以下是一些代碼，比較使用正則表達式進行字段解析的時間，並與str.split進行比較。

首先，我創建了一些假存儲在rows列表中的假測試數據。這樣做使我的演示代碼比從文件讀取數據更簡單，但更重要的是，它消除了文件讀取的開銷，因此我們可以更準確地比較解析速度。

順便說一句，你應該在字段解析循環之外保存sectionNameMatch.group(1)，而不是必須在每個字段行上進行該調用。

首先，我將說明我的代碼正確解析數據。 :)

import re 
from pprint import pprint 
from time import perf_counter 

# Make some test data 
num = 10 
rows = [] 
for i in range(1, num): 
    j = 100 * i 
    rows.append(' field_{:03} {}:{} some_text here ;'.format(i, j, j - 50)) 
rows.append('};') 
print('\n'.join(rows)) 

# Select whether to use regex to do the parsing or `str.split` 
use_regex = True 
print('Testing {}'.format(('str.split', 'regex')[use_regex])) 

fh = iter(rows) 
fieldMap = {} 

nextLine = next(fh) 
start = perf_counter() 
if use_regex: 
    while not "};" in nextLine: 
     fieldMatch = re.search(r'(\S+)\s+(\d+):(\d+)', nextLine) 
     if fieldMatch: 
      fieldMap[fieldMatch.group(1)] = [fieldMatch.group(2), fieldMatch.group(3)] 
     nextLine = next(fh) 
else: 
    while not "};" in nextLine: 
     if nextLine: 
      data = nextLine.split(maxsplit=2) 
      fieldMap[data[0]] = data[1].split(':') 
     nextLine = next(fh) 

print('time: {:.6f}'.format(perf_counter() - start)) 
pprint(fieldMap) 

輸出

field_001 100:50 some_text here ; 
field_002 200:150 some_text here ; 
field_003 300:250 some_text here ; 
field_004 400:350 some_text here ; 
field_005 500:450 some_text here ; 
field_006 600:550 some_text here ; 
field_007 700:650 some_text here ; 
field_008 800:750 some_text here ; 
field_009 900:850 some_text here ; 
}; 
Testing regex 
time: 0.001946 
{'field_001': ['100', '50'], 
'field_002': ['200', '150'], 
'field_003': ['300', '250'], 
'field_004': ['400', '350'], 
'field_005': ['500', '450'], 
'field_006': ['600', '550'], 
'field_007': ['700', '650'], 
'field_008': ['800', '750'], 
'field_009': ['900', '850']} 

下面是與use_regex = False輸出;我不打擾重新打印輸入數據。

Testing str.split 
time: 0.000100 
{'field_001': ['100', '50'], 
'field_002': ['200', '150'], 
'field_003': ['300', '250'], 
'field_004': ['400', '350'], 
'field_005': ['500', '450'], 
'field_006': ['600', '550'], 
'field_007': ['700', '650'], 
'field_008': ['800', '750'], 
'field_009': ['900', '850']} 

現在進行真正的測試。我將設置num = 200000並註釋掉打印輸入數據的行。

Testing regex 
time: 3.640832 

Testing str.split 
time: 2.480094 

正如你所看到的，正則表達式的版本慢了大約50％。

這些時間是在我運行Python 3.6.0的古老的2GHz 32位機器上獲得的，所以你的速度可能不同。 ;）如果你的Python沒有time.perf_counter，你可以改用time.time。