I am trying to read a csv file line by line and only select the 2nd and the 3rd cell from left, and the 3rd cell from the right. For example, if there are 17 cells in this line, I am going to take the 15th cell. Then I want to combine those 3 cells, separated by comma, and then to write this line to a new csv file.
Foe now, I am just using a for loop to access each line and then split them by comma. Then I select the cells I want and combine them as a string and append to a big String variable. Once the for-loop finishes, I write out the file by writeLines(). However, it takes a long time to finish this process because there are 2.8 million rows and it takes a lot of memory. Is there any way to make it more efficient? or can I write the output file line by line in the for-loop?
FileLinebyLine <- read_lines("testfile.csv")
pt<-proc.time()
NewFile <- ""
RowList <- list()
for (i in 1:length(FileLinebyLine))
{
a <- strsplit(FileLinebyLine[i],",")
RowList[i] = paste(a[[1]][2],a[[1]][3],a[[1]][(length(a[[1]]) - 2)], sep = ",")
}
NewFile <- paste(unlist(RowList), sep = "\n")
proc.time()-pt
outputfile <- file("output.txt")
writeLines(NewFile,outputfile)
close(outputfile)
I have also tried to use write_lines() in the for loop but it always gives me the error Error in
isOpen(path) : invalid connection
Can anyone help me? Appreciate that!!!