2011-02-08 148 views
2

我想從文件夾中的所有編碼文件中刪除所有重音符號..我已經在構建文件列表中成功,問題是當我嘗試使用unicodedata進行標準化我得到的錯誤: **回溯(最近一次調用最後一次): 文件「/usr/lib/gedit-2/plugins/pythonconsole/console.py」,第336行,在__run中 exec command in self .namespace 文件 「」,第2行,在 UnicodeDecodeError錯誤:在位置25 'UTF8' 編解碼器不能解碼字節0xf3:無效延續字節 **Python - 從文件夾中的所有文件中刪除重音

if options.remove_nonascii: 
    nERROR = 0 
    print _("# Removing all acentuation from coding files in %s") % (options.folder) 
    exts = ('.f90', '.f', '.cpp', '.c', '.hpp', '.h', '.py'); files=set() 
    for dirpath, dirnames, filenames in os.walk(options.folder): 
     for filename in (f for f in filenames if f.endswith(exts)): 
      files.add(os.path.join(dirpath,filename)) 
    for i in range(len(files)): 
     f = files.pop() ; 
     os.rename(f,f+'.BACK') 
     with open(f,'w') as File: 
      for line in open(f+'.BACK').readlines(): 
       try: 
        newLine = unicodedata.normalize('NFKD',unicode(line)).encode('ascii','ignore') 
        File.write(newLine) 
       except UnicodeDecodeError: 
        nERROR +=1 
        print "ERROR n %i - Could not remove from Line: %i" % (nERROR,i) 
        newLine = line 
        File.write(newLine) 

回答

4

它看起來像文件可能與CP1252編解碼器進行編碼:

In [18]: print('\xf3'.decode('cp1252')) 
ó 

unicode(line)失敗,因爲unicode試圖與utf-8編解碼器解碼line代替,因此錯誤UnicodeDecodeError: 'utf8' codec can't decode...

你可以嘗試先用CP1252解碼line,如果失敗,嘗試UTF-8:

if options.remove_nonascii: 
    nERROR = 0 
    print _("# Removing all acentuation from coding files in %s") % (options.folder) 
    exts = ('.f90', '.f', '.cpp', '.c', '.hpp', '.h', '.py'); files=set() 
    for dirpath, dirnames, filenames in os.walk(options.folder): 
     for filename in (f for f in filenames if f.endswith(exts)): 
      files.add(os.path.join(dirpath,filename)) 
    for i,f in enumerate(files): 
     os.rename(f,f+'.BACK') 
     with open(f,'w') as fout: 
      with open(f+'.BACK','r') as fin: 
       for line fin: 
        try: 
         try: 
          line=line.decode('cp1252') 
         except UnicodeDecodeError: 
          line=line.decode('utf-8') 
          # If this still raises an UnicodeDecodeError, let the outer 
          # except block handle it 
         newLine = unicodedata.normalize('NFKD',line).encode('ascii','ignore') 
         fout.write(newLine) 
        except UnicodeDecodeError: 
         nERROR +=1 
         print "ERROR n %i - Could not remove from Line: %i" % (nERROR,i) 
         newLine = line 
         fout.write(newLine) 

順便說一句,

unicodedata.normalize('NFKD',line).encode('ascii','ignore') 

是有點危險。例如,它消除u'ß」和完全的一些引號:

In [23]: unicodedata.normalize('NFKD',u'ß').encode('ascii','ignore') 
Out[23]: '' 

In [24]: unicodedata.normalize('NFKD',u'‘’「」').encode('ascii','ignore') 
Out[24]: '' 

如果這是一個問題,然後使用unidecode module

In [25]: import unidecode 
In [28]: print(unidecode.unidecode(u'‘’「」ß')) 
''""ss 
+0

謝謝..使用unidecode解決了! – canesin 2011-02-08 17:18:18

1

你可能想使用unicode(line)指定編碼,比如unicode(line,'utf-8')

如果你不知道,sys.getfilesystemencoding()可能是你的朋友。

相關問題