JSON到CSV轉換器-YELP數據集-python

Question

我對數據挖掘感興趣，我想打開並使用yelp的數據。 Yelp的數據是json格式，在它的網站上有以下代碼將json轉換爲csv。但是，當我打開命令行並寫入以下內容：

$ python json_to_csv_converter.py yelp_academic_dataset.json 

我收到一條錯誤消息。你能幫我嗎？

的代碼是：

# -*- coding: utf-8 -*- 
"""Convert the Yelp Dataset Challenge dataset from json format to csv. 
For more information on the Yelp Dataset Challenge please visit http://yelp.com/dataset_challenge 
""" 
import argparse 
import collections 
import csv 
import simplejson as json 


def read_and_write_file(json_file_path, csv_file_path, column_names): 
    """Read in the json dataset file and write it out to a csv file, given the column names.""" 
    with open(csv_file_path, 'wb+') as fout: 
     csv_file = csv.writer(fout) 
     csv_file.writerow(list(column_names)) 
     with open(json_file_path) as fin: 
      for line in fin: 
       line_contents = json.loads(line) 
       csv_file.writerow(get_row(line_contents, column_names)) 

def get_superset_of_column_names_from_file(json_file_path): 
    """Read in the json dataset file and return the superset of column names.""" 
    column_names = set() 
    with open(json_file_path) as fin: 
     for line in fin: 
      line_contents = json.loads(line) 
      column_names.update(
        set(get_column_names(line_contents).keys()) 
        ) 
    return column_names 

def get_column_names(line_contents, parent_key=''): 
    """Return a list of flattened key names given a dict. 
    Example: 
     line_contents = { 
      'a': { 
       'b': 2, 
       'c': 3, 
       }, 
     } 
     will return: ['a.b', 'a.c'] 
    These will be the column names for the eventual csv file. 
    """ 
    column_names = [] 
    for k, v in line_contents.iteritems(): 
     column_name = "{0}.{1}".format(parent_key, k) if parent_key else k 
     if isinstance(v, collections.MutableMapping): 
      column_names.extend(
        get_column_names(v, column_name).items() 
        ) 
     else: 
      column_names.append((column_name, v)) 
    return dict(column_names) 

def get_nested_value(d, key): 
    """Return a dictionary item given a dictionary `d` and a flattened key from `get_column_names`. 

    Example: 
     d = { 
      'a': { 
       'b': 2, 
       'c': 3, 
       }, 
     } 
     key = 'a.b' 
     will return: 2 

    """ 
    if '.' not in key: 
     if key not in d: 
      return None 
     return d[key] 
    base_key, sub_key = key.split('.', 1) 
    if base_key not in d: 
     return None 
    sub_dict = d[base_key] 
    return get_nested_value(sub_dict, sub_key) 

def get_row(line_contents, column_names): 
    """Return a csv compatible row given column names and a dict.""" 
    row = [] 
    for column_name in column_names: 
     line_value = get_nested_value(
         line_contents, 
         column_name, 
         ) 
     if isinstance(line_value, unicode): 
      row.append('{0}'.format(line_value.encode('utf-8'))) 
     elif line_value is not None: 
      row.append('{0}'.format(line_value)) 
     else: 
      row.append('') 
    return row 

if __name__ == '__main__': 
    """Convert a yelp dataset file from json to csv.""" 

    parser = argparse.ArgumentParser(
      description='Convert Yelp Dataset Challenge data from JSON format to CSV.', 
      ) 

    parser.add_argument(
      'json_file', 
      type=str, 
      help='The json file to convert.', 
      ) 

    args = parser.parse_args() 

    json_file = args.json_file 
    csv_file = '{0}.csv'.format(json_file.split('.json')[0]) 

    column_names = get_superset_of_column_names_from_file(json_file) 
    read_and_write_file(json_file, csv_file, column_names) 

錯誤，我在命令行獲得：

Traceback (most recent call last): 
File "json_to_csv_converter.py", line 122, in column_names=get_superset_of_column_names_from_file 
File "json_to_csv_converter.py", line 25, in get_superset_of_column_names_from_file 
for line in fin: 
File "C:\Users\Bengi\Appdata\Local\Programs\Python\Python35-32\lib\encodings\cp1252.py" line 23, in decode 
return codecs.charmap_decode(input, self_errors,decoding_table)[0] 
Unicode Decode Error: 'charmap' codec cant decode byte 0X9d in position 1102: character maps to 

Answer 1

由錯誤信息來看似乎是壞了你的輸入文件。它看起來像json_to_csv_converter.py已確定文件編碼是Windows 1252，但文件中有一個或多個無效字符，即'\x9d'，它不是有效的1252碼點。

檢查您的文件是否正確編碼。我猜想這個文件是UTF8編碼的，但是由於某種原因它正在被處理，就好像它是Windows 1252一樣。你編輯了文件嗎？

Answer 2

你有文件編碼問題。你應該把encoding ='utf8'放在json文件打開的函數之後，例如：with open(json_file_path, encoding='utf8') as fin:

Answer 3

Winzip似乎以某種方式破壞它。我工作圍繞這個由：

• 使用7-Zip的提取tar文件。

• 編輯腳本強制使用UTF-8編碼，像這樣：

  with open(json_file_path, encoding='utf8') as fin: