scipy/sklearn用於文檔分類的稀疏矩陣分解

Question

我試圖在使用標準scikit-learn方法時在大型語料庫（4 mil文檔）上進行文檔分類並且遇到內存錯誤。在清理/填充我的數據後，我有一個非常稀疏的矩陣，大約有1個字。我的第一個想法是使用sklearn.decomposition.TruncatedSVD，但由於內存錯誤，我無法執行足夠大的k的.fit（）操作（我能做的最大操作只佔數據方差的25％ ）。我試着按照sklearn分類here，但是在做KNN分類時仍然記憶猶新。 我想手動進行核心矩陣變換，將矩陣PCA/SVD應用於矩陣以降低維度，但需要先計算特徵向量的方法。我希望能使用scipy.sparse.linalg.eigs 有沒有一種方法來計算特徵向量矩陣，以完成下面顯示的代碼？

from sklearn.feature_extraction.text import TfidfVectorizer 
import scipy.sparse as sp 
import numpy as np 
import cPickle as pkl 
from sklearn.neighbors import KNeighborsClassifier 

def pickleLoader(pklFile): 
    try: 
     while True: 
      yield pkl.load(pklFile) 
    except EOFError: 
     pass 

#sample docs 
docs = ['orange green','purple green','green chair apple fruit','raspberry pie banana yellow','green raspberry hat ball','test row green apple'] 
classes = [1,0,1,0,0,1] 
#first k eigenvectors to keep 
k = 3 

#returns sparse matrix 
tfidf = TfidfVectorizer() 
tfs = tfidf.fit_transform(docs) 

#write sparse matrix to file 
pkl.dump(tfs, open('pickleTest.p', 'wb')) 



#NEEDED - THIS LINE THAT CALCULATES top k eigenvectors 
del tfs 

x = np.empty([len(docs),k]) 

#iterate over sparse matrix 
with open('D:\\GitHub\\Avitro-Classification\\pickleTest.p') as f: 
    rowCounter = 0 
    for dataRow in pickleLoader(f): 
     colCounter = 0 
     for col in k: 
      x[rowCounter, col] = np.sum(dataRow * eingenvectors[:,col]) 
f.close() 

clf = KNeighborsClassifier(n_neighbors=10) 
clf.fit(x, k_class) 

任何援助或指導將不勝感激！如果有更好的方法來做到這一點，我很樂意嘗試一種不同的方法，但我想嘗試KNN這個大的稀疏數據集，最好使用一些降維（這在我運行的小測試數據集上表現得非常好 - 我討厭失去我的性能，因爲愚蠢的內存限制）

編輯：這是我第一次嘗試運行代碼，導致我下來做我自己外的核心稀疏PCA實現的路徑。修復這個內存錯誤的任何幫助都會讓這個更容易！

from sklearn.decomposition import TruncatedSVD 
import pickle 

dataFolder = 'D:\\GitHub\\project\\' 

# in the form of a list: [word sample test word, big sample test word test, green apple test word] 
descWords = pickle.load(open(dataFolder +'descriptionWords.p')) 

vectorizer = TfidfVectorizer() 
X_words = vectorizer.fit_transform(descWords) 

print np.shape(X_words) 

del descWords 
del vectorizer 

svd = TruncatedSVD(algorithm='randomized', n_components=50000, random_state=42) 
output = svd.fit_transform(X_words) 

與輸出：

(3995803, 923633) 
--------------------------------------------------------------------------- 
MemoryError        Traceback (most recent call last) 
<ipython-input-27-c0db86bd3830> in <module>() 
    16 
    17 svd = TruncatedSVD(algorithm='randomized', n_components=50000, random_state=42) 
---> 18 output = svd.fit_transform(X_words) 

C:\Python27\lib\site-packages\sklearn\decomposition\truncated_svd.pyc in fit_transform(self, X, y) 
    173    U, Sigma, VT = randomized_svd(X, self.n_components, 
    174           n_iter=self.n_iter, 
--> 175           random_state=random_state) 
    176   else: 
    177    raise ValueError("unknown algorithm %r" % self.algorithm) 

C:\Python27\lib\site-packages\sklearn\utils\extmath.pyc in randomized_svd(M, n_components, n_oversamples, n_iter, transpose, flip_sign, random_state, n_iterations) 
    297   M = M.T 
    298 
--> 299  Q = randomized_range_finder(M, n_random, n_iter, random_state) 
    300 
    301  # project M to the (k + p) dimensional space using the basis vectors 

C:\Python27\lib\site-packages\sklearn\utils\extmath.pyc in randomized_range_finder(A, size, n_iter, random_state) 
    212 
    213  # generating random gaussian vectors r with shape: (A.shape[1], size) 
--> 214  R = random_state.normal(size=(A.shape[1], size)) 
    215 
    216  # sampling the range of A using by linear projection of r 

C:\Python27\lib\site-packages\numpy\random\mtrand.pyd in mtrand.RandomState.normal (numpy\random\mtrand\mtrand.c:9968)() 

C:\Python27\lib\site-packages\numpy\random\mtrand.pyd in mtrand.cont2_array_sc (numpy\random\mtrand\mtrand.c:2370)() 

MemoryError: 

Answer 1

稀疏數據外的核心SVD或PCA 不執行scikit學習0.15.2。您可能需要嘗試gensim。

編輯：我忘了在我的第一個回覆中指定「關於稀疏數據」。