我已經有一個同事問我從「Yelp數據集挑戰」中將6個巨大文件從有點「平坦」的普通JSON轉換爲CSV (他認爲它們看起來像有趣的教學資料)。Python性能調優:JSON到CSV,大文件
我想我可以一鼓作氣出來:
# With thanks to http://www.diveintopython3.net/files.html and https://www.reddit.com/r/MachineLearning/comments/33eglq/python_help_jsoncsv_pandas/cqkwyu8/
import os
import pandas
jsondir = 'c:\\example\\bigfiles\\'
csvdir = 'c:\\example\\bigcsvfiles\\'
if not os.path.exists(csvdir): os.makedirs(csvdir)
for file in os.listdir(jsondir):
with open(jsondir+file, 'r', encoding='utf-8') as f: data = f.readlines()
df = pandas.read_json('[' + ','.join(map(lambda x: x.rstrip(), data)) + ']')
df.to_csv(csvdir+os.path.splitext(file)[0]+'.csv',index=0,quoting=1)
不幸的是,我的電腦的內存是不達標的任務在這個尺寸的文件。 (即使我擺脫了循環,雖然它在不到一分鐘的時間內甩出了一個50MB的文件,但它努力避免凍結我的電腦或崩潰在100MB +文件上,而最大的文件是3.25GB。)
是否還有其他簡單但性能可以運行的東西?
在循環中會很好,但如果它對內存有影響(只有6個文件),我也可以運行6次w /單獨的文件名。
下面是一個「.json」文件內容的例子 - 注意每個文件實際上有很多JSON對象,每行1個。
{"business_id":"xyzzy","name":"Business A","neighborhood":"","address":"XX YY ZZ","city":"Tempe","state":"AZ","postal_code":"85283","latitude":33.32823894longitude":-111.28948,"stars":3,"review_count":3,"is_open":0,"attributes":["BikeParking: True","BusinessAcceptsBitcoin: False","BusinessAcceptsCreditCards: True","BusinessParking: {'garage': False, 'street': False, 'validated': False, 'lot': True, 'valet': False}","DogsAllowed: False","RestaurantsPriceRange2: 2","WheelchairAccessible: True"],"categories":["Tobacco Shops","Nightlife","Vape Shops","Shopping"],"hours":["Monday 11:0-21:0","Tuesday 11:0-21:0","Wednesday 11:0-21:0","Thursday 11:0-21:0","Friday 11:0-22:0","Saturday 10:0-22:0","Sunday 11:0-18:0"],"type":"business"}
{"business_id":"dsfiuweio2f","name":"Some Place","neighborhood":"","address":"Strip or something","city":"Las Vegas","state":"NV","postal_code":"89106","latitude":36.189134,"longitude":-115.92094,"stars":1.5,"review_count":2,"is_open":1,"attributes":["BusinessAcceptsBitcoin: False","BusinessAcceptsCreditCards: True"],"categories":["Caterers","Grocery","Food","Event Planning & Services","Party & Event Planning","Specialty Food"],"hours":["Monday 0:0-0:0","Tuesday 0:0-0:0","Wednesday 0:0-0:0","Thursday 0:0-0:0","Friday 0:0-0:0","Saturday 0:0-0:0","Sunday 0:0-0:0"],"type":"business"}
{"business_id":"abccb","name":"La la la","neighborhood":"Blah blah","address":"Yay that","city":"Toronto","state":"ON","postal_code":"M6H 1L5","latitude":43.283984,"longitude":-79.28284,"stars":2,"review_count":6,"is_open":1,"attributes":["Alcohol: none","Ambience: {'romantic': False, 'intimate': False, 'classy': False, 'hipster': False, 'touristy': False, 'trendy': False, 'upscale': False, 'casual': False}","BikeParking: True","BusinessAcceptsCreditCards: True","BusinessParking: {'garage': False, 'street': False, 'validated': False, 'lot': False, 'valet': False}","Caters: True","GoodForKids: True","GoodForMeal: {'dessert': False, 'latenight': False, 'lunch': False, 'dinner': False, 'breakfast': False, 'brunch': False}","HasTV: True","NoiseLevel: quiet","OutdoorSeating: False","RestaurantsAttire: casual","RestaurantsDelivery: True","RestaurantsGoodForGroups: True","RestaurantsPriceRange2: 1","RestaurantsReservations: False","RestaurantsTableService: False","RestaurantsTakeOut: True","WiFi: free"],"categories":["Restaurants","Pizza","Chicken Wings","Italian"],"hours":["Monday 11:0-2:0","Tuesday 11:0-2:0","Wednesday 11:0-2:0","Thursday 11:0-3:0","Friday 11:0-3:0","Saturday 11:0-3:0","Sunday 11:0-2:0"],"type":"business"}
嵌套的JSON數據可以簡單地保留爲表示它的字符串文字 - 我只是想將頂級密鑰轉換爲CSV文件標題。
而不是**一次讀取和解析整個文件**,您可以嘗試**一次閱讀一個json字典或一個csv行**,然後解析並插入到csv。這將需要更多的手動編碼,但會在文件流風格下運行良好。 –