如何使用scikit-learn對文本對進行分類？

Question

我已閱讀關於這一主題的許多不同的博客，但一直沒能找到一個明確的解決方案。我有以下情況：

1. 我有一對標籤爲1或-1的文本列表。
2. 對於每個文本對，我想要的功能是在以下方式的連接：）F（= TFIDF（T1）「CONCAT」 TFIDF（T2）

如何做同樣的任何建議？我有以下的代碼，但它給出了一個錯誤：

count_vect = TfidfVectorizer(analyzer=u'char', ngram_range=ngram_range) 
    X0_train_counts = count_vect.fit_transform([x[0] for x in training_documents]) 
    X1_train_counts = count_vect.fit_transform([x[1] for x in training_documents]) 
    combined_features = FeatureUnion([("x0", X0_train_counts), ("x1", X1_train_counts)]) 
    clf = LinearSVC().fit(combined_features, training_target) 
    average_training_accuracy += clf.score(combined_features, training_target) 

這是我得到的錯誤：

--------------------------------------------------------------------------- 
TypeError         Traceback (most recent call last) 
scoreEdgesUsingClassifier(None, pos, neg, 1,ngram_range=(2,5), max_size=1000000, test_size=100000) 

scoreEdgesUsingClassifier(unc, pos, neg, number_of_iterations, ngram_range, max_size, test_size) 
X0_train_counts = count_vect.fit_transform([x[0] for x in training_documents]) 
X1_train_counts = count_vect.fit_transform([x[1] for x in training_documents]) 
combined_features = FeatureUnion([("x0", X0_train_counts), ("x1", X1_train_counts)]) 
print "Done transforming, now training classifier" 

lib/python2.7/site-packages/sklearn/pipeline.pyc in __init__(self, transformer_list, n_jobs, transformer_weights) 
616   self.n_jobs = n_jobs 
617   self.transformer_weights = transformer_weights 
--> 618   self._validate_transformers() 
619 
620  def get_params(self, deep=True): 

lib/python2.7/site-packages/sklearn/pipeline.pyc in _validate_transformers(self) 
660     raise TypeError("All estimators should implement fit and " 
661         "transform. '%s' (type %s) doesn't" % 
--> 662         (t, type(t))) 
663 
664  def _iter(self): 

TypeError: All estimators should implement fit and transform. ' (0, 49025) 0.0575144797079 

(254741, 38401) 0.184394443164 
(254741, 201747) 0.186080393768 
(254741, 179231) 0.195062580945 
(254741, 156925) 0.211367771299 
(254741, 90026) 0.202458920022' (type <class 'scipy.sparse.csr.csr_matrix'>) doesn't 

更新

這裏的解決方案：

count_vect = TfidfVectorizer(analyzer=u'char', ngram_range=ngram_range) 
    training_docs_combined = [x[0] for x in training_documents] + [x[1] for x in training_documents]   
    X_train_counts = count_vect.fit_transform(training_docs_combined) 
    concat_features = hstack((X_train_counts[0:len(training_docs_combined)/2 ], X_train_counts[len (training_docs_combined)/2:])) 

    clf = LinearSVC().fit(concat_features, training_target) 
    average_training_accuracy += clf.score(concat_features, training_target) 

Answer 1

FeatureUnion從scikit學習作爲輸入估計量，不是數據陣列。

您可以串連產生的X0_train_counts，X1_train_counts陣列簡單地用scipy.sparse.hstack，或創建TfidfVectorizer兩個獨立的情況下，適用FeatureUnion給他們，然後調用fit_transform方法。