使用Python創建語料庫

Question

我是Python新手， 我使用R創建了一個術語文檔矩陣，我想了解如何使用Python創建它。

我正在讀取數據框Res_Desc_Train中可用的Description列中的文本數據。但不知道如何使用在python中創建文檔術語矩陣的功能，如果有任何有助於學習的文檔，這將會很有幫助。

下面是代碼，我在R.

docs <- Corpus(VectorSource(Res_Desc_Train$Description)) 
docs <-tm_map(docs,content_transformer(tolower)) 

#remove potentially problematic symbols 
toSpace <- content_transformer(function(x, pattern) { return (gsub(pattern, " ", x))}) 
removeSpecialChars <- function(x) gsub("[^a-zA-Z0-9 ]","",x) 
docs <- tm_map(docs, toSpace, "/") 
docs <- tm_map(docs, toSpace, "-") 
docs <- tm_map(docs, toSpace, ":") 
docs <- tm_map(docs, toSpace, ";") 
docs <- tm_map(docs, toSpace, "@") 
docs <- tm_map(docs, toSpace, "\\(") 
docs <- tm_map(docs, toSpace, ")") 
docs <- tm_map(docs, toSpace, ",") 
docs <- tm_map(docs, toSpace, "_") 
docs <- tm_map(docs, content_transformer(removeSpecialChars)) 
docs <- tm_map(docs, content_transformer(tolower)) 
docs <- tm_map(docs, removeWords, stopwords("en")) 
docs <- tm_map(docs, removePunctuation) 
docs <- tm_map(docs, stripWhitespace) 
docs <- tm_map(docs, removeNumbers) 

#inspect(docs[440]) 
dataframe<-data.frame(text=unlist(sapply(docs, `[`, "content")), stringsAsFactors=F) 

BigramTokenizer <- 
    function(x) 
    unlist(lapply(ngrams(words(x), 2), paste, collapse = " "), use.names = FALSE) 

dtm <- DocumentTermMatrix(docs,control=list(stopwords=FALSE,wordLengths =c(2,Inf),tokenize = BigramTokenizer)) 

Weighteddtm <- weightTfIdf(dtm,normalize=TRUE) 
mat.df <- as.data.frame(data.matrix(Weighteddtm), stringsAsfactors = FALSE) 
mat.df <- cbind(mat.df, Res_Desc_Train$Group) 
colnames(mat.df)[ncol(mat.df)] <- "Group" 
Assignment.Distribution <- table(mat.df$Group) 

Res_Desc_Train_Assign <- mat.df$Group 

Assignment.Distribution <- table(mat.df$Group) 

### Feature has different ranges, normalizing to bring ranges from 0 to 1 
### Another way to standardize using z-scores 

normalize <- function(x) { 
    y <- min(x) 
    z <- max(x) 
    temp <- x - y 
    temp1 <- (z - y) 
    temp2 <- temp/temp1 
    return(temp2) 
} 
#normalize(c(1,2,3,4,5)) 

num_col <- ncol(mat.df)-1 
mat.df_normalize <- as.data.frame(lapply(mat.df[,1:num_col], normalize)) 
mat.df_normalize <- cbind(mat.df_normalize, Res_Desc_Train_Assign) 
colnames(mat.df_normalize)[ncol(mat.df_normalize)] <- "Group" 

Answer 1

通常，當您需要處理在Python文本的最佳工具是NLTK使用。在你的具體情況中，有一個特定的python包創建term-document-matrix。這個軟件包被稱爲Textmining。

此外，如果你需要使用正則表達式，你可以使用python的re包。否則，您可以直接使用標記器來構建NLTK。