我有一個csv文件有500萬行。 我想將文件拆分成由用戶指定的行數。通過python分割一個大的csv文件
已開發了以下代碼,但其執行時間太長。任何人都可以幫助我優化代碼。
import csv
print "Please delete the previous created files. If any."
filepath = raw_input("Enter the File path: ")
line_count = 0
filenum = 1
try:
in_file = raw_input("Enter Input File name: ")
if in_file[-4:] == ".csv":
split_size = int(raw_input("Enter size: "))
print "Split Size ---", split_size
print in_file, " will split into", split_size, "rows per file named as OutPut-file_*.csv (* = 1,2,3 and so on)"
with open (in_file,'r') as file1:
row_count = 0
reader = csv.reader(file1)
for line in file1:
#print line
with open(filepath + "\\OutPut-file_" +str(filenum) + ".csv", "a") as out_file:
if row_count < split_size:
out_file.write(line)
row_count = row_count +1
else:
filenum = filenum + 1
row_count = 0
line_count = line_count+1
print "Total Files Written --", filenum
else:
print "Please enter the Name of the file correctly."
except IOError as e:
print "Oops..! Please Enter correct file path values", e
except ValueError:
print "Oops..! Please Enter correct values"
我自己也嘗試沒有 「打開」
來些更傳統的單位超過萬盧比?;) – liborm
關於與不同的文件指針尋求不同點,並使用所有這些通過並行協同例程/ GEVENT什麼? – SRC
我還沒有嘗試過..你可以請幫助相同。多線程或多任務在這裏會有幫助。 – user2597209