虔誠分析 - 交叉驗證無效分數

Question

我正在使用NLTK和SKlearn測試一個輿情分析模型。

Movie_reviews數據具有「pos」和「neg」標籤。對於使用「特徵集」的分類器進行訓練。我正在對訓練數據和測試數據的準確性進行交叉驗證。然而，交叉驗證總是遠高於準確性。在下面的邏輯迴歸算法CV = 97（平均），準確度= 70的例子中，我已經測試了其他算法，並且仍然交叉驗證非常高。

我很確定我用於交叉驗證的代碼是不正確的。

import nltk 
import random 
from nltk.corpus import movie_reviews 
from sklearn import cross_validation 
from nltk.classify.scikitlearn import SklearnClassifier 
from sklearn.linear_model import LogisticRegression, SGDClassifier 

documents = [(list(movie_reviews.words(fileid)), category) 
      for category in movie_reviews.categories() 
      for fileid in movie_reviews.fileids(category)] 
random.shuffle(documents) 

all_words = [] 
for w in movie_reviews.words(): 
    all_words.append(w.lower()) 

all_words = nltk.FreqDist(all_words) 
word_features = list(all_words.keys())[:3000] 

def find_features(document): 
    words = set(document) 
    features = {} 
    for w in word_features: 
     features[w] = (w in words) 
    return features 

featuresets = [(find_features(rev), category) for (rev, category) in documents]   
training_set = featuresets[:1500] 
testing_set = featuresets[1500:] 

cv = cross_validation.KFold(len(training_set), n_folds=10, shuffle=True, random_state=None) 
LogisticRegression_classifier = SklearnClassifier(LogisticRegression())  
for traincv, testcv in cv: 
    classifier = LogisticRegression_classifier.train(training_set[traincv[0]:traincv[len(traincv)-1]]) 
    print ('CV_accuracy:', nltk.classify.util.accuracy(classifier, training_set[testcv[0]:testcv[len(testcv)-1]])) 

print("LogisticRegression_classifier accuracy percent:", (nltk.classify.accuracy(LogisticRegression_classifier, testing_set))*100) 

Answer 1

您正在使用training_set [traincv [0]：traincv [長度（traincv）-1]]這意味着從traincv範圍[0]到traincv [長度（traincv）-1]

在你的情況下，traincv [0]和testcv [0]將總是接近0，traincv [len（traincv）-1]和testcv [len（testcv）-1]將接近1499.所以，你幾乎使用在進行N-FOLD驗證時用於訓練和測試的相同數據。

在這裏，您實際上需要使用traincv和testcv中的子集索引。

import numpy as np 
training_set = np.array(training_set) 
for traincv, testcv in cv: 
    classifier = LogisticRegression_classifier.train(training_set[traincv]) 
    print ('CV_accuracy:', nltk.classify.util.accuracy(classifier, training_set[testcv]