如何從Python中的文本數據文件中提取數據子集

Question

我有一個文本文件，其中相關數據（行x列）僅出現在「開始」和「結束」關鍵字之間。見下文。我想編寫一個可以提取這些數據子集的代碼。如果一行以「開始」開始，然後是數據，但沒有後續的「結束」關鍵字，那麼我想忽略這些數據。在我的例子中，data1和data3是相關的，但data2並不是因爲它沒有被「開始」和「結束」關鍵字所包圍。

start 
data1 (matrix of data) /relevant because data1 is enclosed by "start" and "end" 
end 
start 
data2 (matrix of data) /not relevant because there is no "end" 
. 
start 
data3 (matrix of data) /relevant for same reason as for data1 
end 
. 
. 
and so on 

我以爲我可以這樣開始：

with open(file_path,'r') as file: 

    text = file.readlines() 
    start_indexes = [] 
    end_indexes = [] 

    for i, line in enumerate(text): 
     if line.startswith('start'): 
      start_indexes.append(i) 
     elif line.startswith('end'): 
      end_indexes.append(i) 

    for i in range(len(start_indexes)): 
     for j in range(len(end_indexes)): 
      if (start_indexes[i] < end_indexes[j] < start_indexes[i+1]): 
       print(start_indexes[i],end_indexes[j]) 

上面的代碼給了我兩個起始行號和結束行號在有相關數據。這是我有點卡住的地方。我現在如何提取相關數據？在我下面的例子中，它將是data1，data3。我是否以「正確」的方式處理問題？我應該使用熊貓嗎？有沒有更高效和直接的方法？

Answer 1

嵌套循環？

您正在經歷開始和結束範圍的每種組合。你只需要那些對應於相同數據的那些數據。

像這樣的東西替換你的for循環：

for start, end in zip(start_indexes, end_indexes): 
    print(text[start + 1:end]) 

zip(a, b, ...)返回與a, b, ...列，根本上給[(a[0], b[0], ...), (a[1], b[1], ...), ...]一個新的列表。通過start_indexes, end_indexes的每一列，給出相應的開始和結束值，然後使用列表片段訪問來獲取這些行的數據。

Answer 2

我會通過只讀順序讀取文件來使用另一種方式（假設「開始」 - 「結束」 - 數據塊中的數據不會太大）。我將創建一個data變量來收集當前塊的數據（無論是否相關）以及具有狀態轉換的變量。

一些僞的Python：

if state == OUTSIDE_BLOCK and line.startswith('start'): 
    state = INSIDE_BLOCK 
elif state == INSIDE_BLOCK and <line contains data>: 
    <Add to data variable> 
elif state == INSIDE_BLOCK and line.startswith('end'): 
    state = OUTSIDE_BLOCK 
    <Process collected data> 
elif state == INSIDE_BLOCK and line.startswith('start'): 
    <Throw away possibly collected data because it was irrelevant> 

Answer 3

我個人覺得正則表達式是處理這種情況的最好辦法：

import re woof0='''start data1 (matrix of data) /relevant because data1 is enclosed by "start" and "end" end start data2 (matrix of data) /not relevant because there is no "end" . start data3 (matrix of data) /relevant for same reason as for data1 end . . and so on ''' re.findall(r'start(\sdata.*|\Sdata.*)\nend',woof0)

輸出：

['\ndata1 (matrix of data) /relevant because data1 is enclosed by "start" and "end"', '\ndata3 (matrix of data) /relevant for same reason as for data1']

Answer 4

設置：

s = '''start 
data1 (matrix of data) /relevant because data1 is enclosed by "start" and "end" 
end 
start 
data2 (matrix of data) /not relevant because there is no "end" 
start 
data3 (matrix of data) /relevant for same reason as for data1 
end 
start 
data4 blah 
''' 
import io 
f = io.StringIO(s) 

對文件進行迭代一次，測試每行的開頭;找出將有效數據塊放入子列表並將它們附加到結果列表中所需的邏輯...

result = [] 
sub = [] 

for line in f: 
    if line.startswith('start'): 
     # possible new data block 
     if sub: 
      # if it isn't empty it must contain 
      # a start --> data block with no end 
      result.append(sub) 
      sub = [] 
     sub = [line] 
    elif line.startswith('end'): 
     # start over 
     sub = [] 
    elif line.startswith('data'): 
     sub.append(line) 
    else: 
     # for lines that don't startwith data, start or end - if any 
     sub.append(line) 

if sub: 
    # start --> data --> EOF or end of string 
    result.append(sub)