如何在Python中逐個讀取文件中的令牌？

Question

我遇到的問題是，在我的代碼中，我無法獲取單個詞/標記以匹配停用詞從原始文本中刪除。相反，我得到了一整句話，因此無法將它與停用詞相匹配。請告訴我一種方法，我可以獲取個人令牌，然後用停用詞匹配並刪除它們。請幫幫我。

from nltk.corpus import stopwords 
import string, os 
def remove_stopwords(ifile): 
    processed_word_list = [] 
    stopword = stopwords.words("urdu") 
    text = open(ifile, 'r').readlines() 
    for word in text: 
     print(word) 
     if word not in stopword: 
       processed_word_list.append('*') 
       print(processed_word_list) 
       return processed_word_list 

if __name__ == "__main__": 
    print ("Input file path: ") 
    ifile = input() 
    remove_stopwords(ifile) 

Answer 1

試試這個：

from nltk.corpus import stopwords 
from nltk.tokenize import word_tokenize 
import string, os, ast 
def remove_stopwords(ifile): 
    processed_word_list = [] 
    stopword = stopwords.words("urdu") 
    words = ast.literal_eval(open(ifile, 'r').read()) 
    for word in words: 
     print(word) 
     if word not in stopword: 
      processed_word_list.append('*') 
     else: 
      processed_word_list.append(word) 
    print(processed_word_list) 
    return processed_word_list 

if __name__ == "__main__": 
    print ("Input file path: ") 
    ifile = input() 
    remove_stopwords(ifile)