2013-04-26 86 views
1

如何最好地分割列表中單詞的向量?這是我目前正在做的事情(感謝geektrader的回答超過here),但它使RStudio顫抖並凍結了很多。這個問題與我以前的問題密切相關。這裏有什麼新的列表結構,這更接近我的實際使用情況。每隔n個單詞將單詞向量拆分(向量在列表中)

# reproducible data 
examp1 <- "When discussing performance with colleagues, teaching, sending a bug report or searching for guidance on mailing lists and here on SO, a reproducible example is often asked and always helpful. What are your tips for creating an excellent example? How do you paste data structures from r in a text format? What other information should you include? Are there other tricks in addition to using dput(), dump() or structure()? When should you include library() or require() statements? Which reserved words should one avoid, in addition to c, df, data, etc? How does one make a great r reproducible example?" 
examp2 <- "Sometimes the problem really isn't reproducible with a smaller piece of data, no matter how hard you try, and doesn't happen with synthetic data (although it's useful to show how you produced synthetic data sets that did not reproduce the problem, because it rules out some hypotheses). Posting the data to the web somewhere and providing a URL may be necessary. If the data can't be released to the public at large but could be shared at all, then you may be able to offer to e-mail it to interested parties (although this will cut down the number of people who will bother to work on it). I haven't actually seen this done, because people who can't release their data are sensitive about releasing it any form, but it would seem plausible that in some cases one could still post data if it were sufficiently anonymized/scrambled/corrupted slightly in some way. If you can't do either of these then you probably need to hire a consultant to solve your problem" 
examp3 <- "You are most likely to get good help with your R problem if you provide a reproducible example. A reproducible example allows someone else to recreate your problem by just copying and pasting R code. There are four things you need to include to make your example reproducible: required packages, data, code, and a description of your R environment. Packages should be loaded at the top of the script, so it's easy to see which ones the example needs. The easiest way to include data in an email is to use dput() to generate the R code to recreate it. For example, to recreate the mtcars dataset in R, I'd perform the following steps: Run dput(mtcars) in R Copy the output In my reproducible script, type mtcars <- then paste. Spend a little bit of time ensuring that your code is easy for others to read: make sure you've used spaces and your variable names are concise, but informative, use comments to indicate where your problem lies, do your best to remove everything that is not related to the problem. The shorter your code is, the easier it is to understand. Include the output of sessionInfo() as a comment. This summarises your R environment and makes it easy to check if you're using an out-of-date package. You can check you have actually made a reproducible example by starting up a fresh R session and pasting your script in. Before putting all of your code in an email, consider putting it on http://gist.github.com/. It will give your code nice syntax highlighting, and you don't have to worry about anything getting mangled by the email system." 
examp4 <- "Do your homework before posting: If it is clear that you have done basic background research, you are far more likely to get an informative response. See also Further Resources further down this page. Do help.search(keyword) and apropos(keyword) with different keywords (type this at the R prompt). Do RSiteSearch(keyword) with different keywords (at the R prompt) to search R functions, contributed packages and R-Help postings. See ?RSiteSearch for further options and to restrict searches. Read the online help for relevant functions (type ?functionname, e.g., ?prod, at the R prompt) If something seems to have changed in R, look in the latest NEWS file on CRAN for information about it. Search the R-faq and the R-windows-faq if it might be relevant (http://cran.r-project.org/faqs.html) Read at least the relevant section in An Introduction to R If the function is from a package accompanying a book, e.g., the MASS package, consult the book before posting. The R Wiki has a section on finding functions and documentation" 
examp5 <- "Before asking a technical question by e-mail, or in a newsgroup, or on a website chat board, do the following: Try to find an answer by searching the archives of the forum you plan to post to. Try to find an answer by searching the Web. Try to find an answer by reading the manual. Try to find an answer by reading a FAQ. Try to find an answer by inspection or experimentation. Try to find an answer by asking a skilled friend. If you're a programmer, try to find an answer by reading the source code. When you ask your question, display the fact that you have done these things first; this will help establish that you're not being a lazy sponge and wasting people's time. Better yet, display what you have learned from doing these things. We like answering questions for people who have demonstrated they can learn from the answers. Use tactics like doing a Google search on the text of whatever error message you get (searching Google groups as well as Web pages). This might well take you straight to fix documentation or a mailing list thread answering your question. Even if it doesn't, saying 「I googled on the following phrase but didn't get anything that looked promising」 is a good thing to do in e-mail or news postings requesting help, if only because it records what searches won't help. It will also help to direct other people with similar problems to your thread by linking the search terms to what will hopefully be your problem and resolution thread. Take your time. Do not expect to be able to solve a complicated problem with a few seconds of Googling. Read and understand the FAQs, sit back, relax and give the problem some thought before approaching experts. Trust us, they will be able to tell from your questions how much reading and thinking you did, and will be more willing to help if you come prepared. Don't instantly fire your whole arsenal of questions just because your first search turned up no answers (or too many). Prepare your question. Think it through. Hasty-sounding questions get hasty answers, or none at all. The more you do to demonstrate that having put thought and effort into solving your problem before seeking help, the more likely you are to actually get help. Beware of asking the wrong question. If you ask one that is based on faulty assumptions, J. Random Hacker is quite likely to reply with a uselessly literal answer while thinking Stupid question..., and hoping the experience of getting what you asked for rather than what you needed will teach you a lesson." 

# make a big list of character vectors containing words 
list_examps <- lapply(1:5, function(i) eval(parse(text=paste0("examp",i)))) 
list_examps <- rep(list_examps, 2000) 

# my current method 
n <- 30 # number of words in each chunk 
temp1 <- vector("list", length(list_examps)) 
temp2 <- vector("list", length(list_examps)) 
for(i in 1:length(list_examps)) 
{ 
    temp1[[i]] <- unlist(strsplit(list_examps[[i]], " ")) 
    temp2[[i]] <- split(unlist(strsplit(temp1[[i]] , " ")), 
         seq_along(unlist(strsplit(temp1[[i]], " ")))%/%n) 
} 
listofnwords <- unlist(temp2, recursive = FALSE) # desired output 

有沒有更有效的方法來做到這一點?

UPDATE 1加入(相當溫和的)機器規格

> sessionInfo() 
    R version 3.0.0 (2013-04-03) 
    Platform: x86_64-w64-mingw32/x64 (64-bit) 

    locale: 
    [1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 
    [3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C       
    [5] LC_TIME=English_United States.1252  

    attached base packages: 
    [1] stats  graphics grDevices utils  datasets methods base  

    loaded via a namespace (and not attached): 
    [1] tools_3.0.0 

    > gc() 
       used (Mb) gc trigger (Mb) max used (Mb) 
    Ncells 434928 23.3  818163 43.7 667722 35.7 
    Vcells 7086406 54.1 12291671 93.8 12291671 93.8 

> Sys.getenv() # relevant exerpts, far off to the right for some reason... 

                                                                                                                                                      NUMBER_OF_PROCESSORS 
                                                                                                                                                          "4" 
                                                                                                                                                          OS 
                                                                                                                                                        "Windows_NT" 

                                                                                                                                                     PROCESSOR_ARCHITECTURE 
                                                                                                                                                         "AMD64" 
                                                                                                                                                      PROCESSOR_IDENTIFIER 
                                                                                                                                              "Intel64 Family 6 Model 42 Stepping 7, GenuineIntel" 
                                                                                                                                                       PROCESSOR_LEVEL 
                                                                                                                                                          "6" 
                                                                                                                                                      PROCESSOR_REVISION 
                                                                                                                                                         "2a07" 

UPDATE 2次答案的速度測試,使用上述數據上的EC2實例(自由層)。贏家是... SimonO101!感謝大家的幫助。

# make a big list of character vectors containing words 
list_examps <- lapply(1:5, function(i) eval(parse(text=paste0("examp",i)))) 
list_examps <- rep(list_examps, 200) 

# my current method 
n <- 30 # number of words in each chunk 
temp1 <- vector("list", length(list_examps)) 
temp2 <- vector("list", length(list_examps)) 

me <- function(list_examps){ 
    for(i in 1:length(list_examps)) 
    { 
    temp1[[i]] <- unlist(strsplit(list_examps[[i]], " ")) 
    temp2[[i]] <- split(unlist(strsplit(temp1[[i]] , " ")), 
         seq_along(unlist(strsplit(temp1[[i]], " ")))%/%n) 
    } 
    listofnwords <- unlist(temp2, recursive = FALSE) # desired output 
} 

dr <- function(list_examps){ 
    f <- function(list_examps) { 
    y <- unlist(strsplit(list_examps, " ")) 
    ly <- length(y) 
    split(y, gl(ly%/%n+1, n, ly)) 
    } 

    listofnwords <- sapply(list_examps, f) 
    listofnwords <- unlist(listofnwords, recursive=F) 
} 

si <- function(list_examps){ 

    words <- unlist((sapply(list_examps , strsplit , " "))) 
    results <- lapply(seq(0, length(words) , by = n) , function(x) c(words[(x+1):(x+n)])) 

} 

er <- function(x){ 
    x <- do.call(paste, x) 
    x <- strsplit(x, ' ')[[1]] 
    result <- split(x, cut(seq_along(x), breaks = seq(0, by = n, length(x)) , include.lowest = TRUE)) 
} 


library(rbenchmark) 

benchmark(
    me(list_examps), 
    dr(list_examps), 
    si(list_examps), 
    er(list_examps), 
    replications = 10) 

而且這裏的結果:

   test replications elapsed relative user.self sys.self user.child sys.child 
2 dr(list_examps)   10 48.104 1.119 47.907 0.000   0   0 
4 er(list_examps)   10 71.316 1.660 70.645 0.568   0   0 
1 me(list_examps)   10 48.156 1.121 48.543 0.000   0   0 
3 si(list_examps)   10 42.971 1.000 42.875 0.000   0   0 
+0

什麼是您的系統規格?在我的桌面上,我能夠在6.44秒內運行上面的所有代碼,幾乎沒有明顯的凍結 – 2013-04-26 09:05:49

+0

也許我應該編輯我的示例數據,使其更像我的真實數據,這些數據是我想要的5000-7000個字左右的矢量分成1000個字塊。 – Ben 2013-04-26 09:22:27

+0

@本人編輯我的答案,現在它應該工作。您的示例數據運行時間爲1.5秒,而您的方法運行時間爲7秒。 – 2013-04-26 10:25:12

回答

2

一個可能的解決方案:

f <- function(x) { 
    y <- unlist(strsplit(x, " ")) 
    ly <- length(y) 
    split(y, gl(ly%/%n+1, n, ly)) 
    } 

listofnwords <- sapply(list_examps, f) 
listofnwords <- unlist(listofnwords, recursive=F) 
+0

儘管這稍微慢一點,但它會返回帶有項目名稱的列表我需要的格式,所以這是一個更完整的答案。 – Ben 2013-04-27 08:06:33

2

我會使用sapplyunlist得到的話,那麼使用lapply將它們串聯到列表中的每30個字的項目(這需要1.6秒在您的示例):

f1 <- function(x){ 

    words <- unlist((sapply(list_examps , strsplit , " "))) 
    results <- lapply(seq(0, length(words) , by = 30) , function(x) c(words[(x+1):(x+30)])) 

} 

而在與其他解決方案,它被定義爲比較:

f2 <- function(x){ 
    x <- do.call(paste, x) 
    x <- strsplit(x, ' ')[[1]] 
    result <- split(x, cut(seq_along(x), breaks = seq(1, by = 30, length(x)) , include.lowest = TRUE)) 
} 

我們看到,在這個例子中是使用lapply和sapply 將運行約兩倍的速度(但取決於你的使用情況可能發生變化):

microbenchmark(f1(list_examps) , f2(list_examps) , times = 10L) 
#Unit: seconds 
#   expr  min  lq median  uq  max neval 
# f1(list_examps) 1.463855 1.531033 1.596762 1.667241 1.815084 10 
# f2(list_examps) 2.787036 2.884537 2.945287 3.019572 3.357706 10 
+0

我不太明白在哪裏放置'n'在代碼中......塊在哪裏產生? – Ben 2013-04-26 09:05:24

+1

啊,對不起 - 我有罪DR(你的帖子不是TL)。讓我重新考慮.... :-) – 2013-04-26 09:07:47

+0

感謝您花時間編輯其他答案並進行比較! – Ben 2013-04-26 23:42:23

3

這個怎麼樣?

x <- do.call(paste, list_examps) 
x <- strsplit(x, ' ')[[1]] 
result <- split(x, cut(seq_along(x), breaks = seq(0, by = 30, length(x)))) 
result[[1]] 
+0

+1不錯的解決方案,但你需要添加'include.lowest = TRUE'來獲得正確的結果 – 2013-04-26 09:27:50

+1

而且我注意到結果中的第一個列表項是31個項目(因爲需要使用include.lowest = TRUE「,但是你可以通過將'seq'的開始更改爲0來解決這個問題,因爲R從1開始索引,然後刪除'include.lowest = TRUE' – 2013-04-26 09:47:45