R: How to combine 300 1GB .rds files into 1 large rds file without reading them into memory?

user_1134241 0
Viewed 693

I have 300+ .rds files each with the same column names and would like to bind them together into a single compressed .rds file that I can transfer via sftp.

Is there a way of doing this efficiently without reading them into memory?

At the moment I'm working with the following code, but this maxes out the memory before writing the file. Any thoughts are much appreciated.

library(tidyverse)
library(data.table)

df <- list.files(pattern = ".rds") %>%
         map(readRDS) %>% 
         data.table::rbindlist()

    saveRDS(df,"df.rds")

Eventually I read in one by one and used read::write_csv("name.csv",append=TRUE) to append them on disk. After that I use {disk.frame} or SQL database to process the data.

r
2 Answers

As others have commented, it probably isn't possible (or at least very difficult) to append/merge .rds files. However, if they are simple columns then there should be on problem converting them to .csv. In which case they can be appended, assuming (as you have said) they have matching column names.

This snippet reads from a list of .rds filenames and appends their data to a .csv. I've next to no R experience, so I'm not sure of how the underlying resources are managed, but in principal this method should allow you to read only one file at a time and thus keep your memory consumption at ~1GB while building your ~300GB .csv.

fileNames <- list('test-one.rds', 'test-two.rds')

for (fileName in fileNames)
{
    rds <- readRDS(fileName)
    for (row in rds){
        write(row, file = 'out.csv', append = TRUE)
    }
}
user_46326270

If your final purpose is to transfer the data, read and write the data in sql server.

You can convert the .rds to .csv and use HeidiSQL for a quick import into a table structured as your csv.

Then on the other end, you can read the data from SQL and convert them into .rds again or just send the .csv over.

user_77306160
Related