R: How do I append data frames to a list?

user_13714796 0
Viewed 325

I have a for loop that reads a list of .csv files, cleans them, and exports the output as a new .csv file. Before outputting the .csv file, I want to add the clean data frame to a list of some sort, so that at the end of the process (there are about 36 .csv files I am cleaning inside the for loop) I can combine all of the cleaned data frames into one single large data frame to be exported as a .csv.

Here is an example of the empty list I initialized at the beginning of the code (before the loop):

csv_list <- vector(mode = "list", length = 36)

When I run the loop, the data frames are not appended as intended. Instead of adding onto the existing columns (the data has the same row and column names, just different values) the list is simply filled with the column names in place of the rows, and vice versa.

Am I using lists incorrectly? Another thought I had was to create an empty data frame and simply merge the cleaned data frames to the empty one each time the loop runs through.

I have no preference for base R, dplyr, etc. Any help would be appreciated.

dplyr dataframe csv r list
1 Answers

Congrats on choosing a good way to iteratively add frames to an object :-)

Two ways, depending on how you are working:

csv_list <- vector(mode = "list", length = 3)
csv_list[[1]] <- mtcars[1:2,]
csv_list[[2]] <- mtcars[1:2,]
csv_list
# [[1]]
#               mpg cyl disp  hp drat    wt  qsec vs am gear carb
# Mazda RX4      21   6  160 110  3.9 2.620 16.46  0  1    4    4
# Mazda RX4 Wag  21   6  160 110  3.9 2.875 17.02  0  1    4    4
# [[2]]
#               mpg cyl disp  hp drat    wt  qsec vs am gear carb
# Mazda RX4      21   6  160 110  3.9 2.620 16.46  0  1    4    4
# Mazda RX4 Wag  21   6  160 110  3.9 2.875 17.02  0  1    4    4
# [[3]]
# NULL

or

csv_list <- list()
csv_list <- c(csv_list, list(mtcars[1:2,]))
csv_list <- c(csv_list, list(mtcars[1:2,]))
csv_list
# [[1]]
#               mpg cyl disp  hp drat    wt  qsec vs am gear carb
# Mazda RX4      21   6  160 110  3.9 2.620 16.46  0  1    4    4
# Mazda RX4 Wag  21   6  160 110  3.9 2.875 17.02  0  1    4    4
# [[2]]
#               mpg cyl disp  hp drat    wt  qsec vs am gear carb
# Mazda RX4      21   6  160 110  3.9 2.620 16.46  0  1    4    4
# Mazda RX4 Wag  21   6  160 110  3.9 2.875 17.02  0  1    4    4

Notice that this last one is open-ended, its length is not necessarily what you need it to be in the end. Even if you pre-allocate length 36 as in your code, if you try to assign [[37]] (first example) or append a 37th frame (second example), it will happily work, there is not bounds-checking in this use-case.

(BTW: unlike data.frames and some other objects, arbitrarily appending to list objects does not scale poorly. For instance, if you tracemem a frame and append a row, you'll see a memory-shift indicating the copy of all of the frame's data (regardless of how many rows you appended. Counter to that, though, if you tracemem(csv_list), you can append to it efficiently with either of the above methods, and the memory address of the list never changes, suggesting storage adjustment is a bit more efficient. That's not to say that it's invulnerable, but it's generally quite good.)

user_33582720
Related