如何使用R將大型CSV數據文件分解爲單個數據文件？

Question

我有一個CSV文件，其第一行包含變量名稱，其餘的行包含數據。有什麼方法可以把它分解成每個只包含R中一個變量的文件？這個解決方案會變得強大嗎？例如。如果輸入文件大小爲100G會怎樣？

輸入文件看起來像

var1,var2,var3 
1,2,hello 
2,5,yay 
... 

我要創建3（或包含很多變量）的文件var1.csv，var2.csv，var3.csv ，使文件類似於 文件1

var1 
1 
2 
... 

文件2

var2? 
2 
5 
... 

文件3

var3 
hello 
yay 

我在Python（How to break a large CSV data file into individual data files?）的解決方案，但我不知道是否R可以做同樣的事情？ Python代碼必須逐行讀取csv文件，然後逐行寫入一行。 R可以做同樣的事嗎？命令read.csv一次讀取整個文件，這可以減慢整個過程。另外它不能讀取100G文件並處理它，因爲R試圖將整個文件讀入內存。我無法在R中找到一個命令，讓您逐行讀取一個csv文件。請幫忙。謝謝！！

Answer 1

您可以將scan，然後write一次一行地寫入一個或多個文件。

i <- 0 
while({x <- scan("file.csv", sep = ",", skip = i, nlines = 1, what = "character"); 
     length(x) > 1}) { 
    write(x[1], "file1.csv", sep = ",", append = T) 
    write(x[2], "file2.csv", sep = ",", append = T) 
    write(x[3], "file3.csv", sep = ",", append = T) 
    i <- i + 1 
} 

編輯!!我正在使用上述數據，複製1000次以上。當我們始終打開文件連接時，我已經對速度進行了比較。

ver1 <- function() { 
    i <- 0 
    while({x <- scan("file.csv", sep = ",", skip = i, nlines = 1, what = "character"); 
     length(x) > 1}) { 
    write(x[1], "file1.csv", sep = ",", append = T) 
    write(x[2], "file2.csv", sep = ",", append = T) 
    write(x[3], "file3.csv", sep = ",", append = T) 
    i <- i + 1 
    } 
} 

system.time(ver1()) # w/ close to 3K lines of data, 3 columns 
## user system elapsed 
## 2.809 0.417 3.629 

ver2 <- function() { 
    f <- file("file.csv", "r") 
    f1 <- file("file1.csv", "w") 
    f2 <- file("file2.csv", "w") 
    f3 <- file("file3.csv", "w") 
    while({x <- scan(f, sep = ",", skip = 0, nlines = 1, what = "character"); 
     length(x) > 1}) { 
    write(x[1], file = f1, sep = ",", append = T, ncol = 1) 
    write(x[2], file = f2, sep = ",", append = T, ncol = 1) 
    write(x[3], file = f3, sep = ",", append = T, ncol = 1) 
    } 
    closeAllConnections() 
} 

system.time(ver2()) 
## user system elapsed 
## 0.257 0.098 0.409