How to read and write csv files in parallel in R

user_8229029 0
Viewed 673

I have several million small (couple MB each) csv files I need to read in, modify, and then write out. How do I do this in parallel in R? I've tried the following, but nothing is happening. I do not need to combine anything - I just want to be able to deal with more than 1 file at a time. I hope this is possible, otherwise, I'm in for a long run. Maybe this isn't even possible - is it possible to read and write more than 1 file from a hard drive at a time? I'm running Windows 10 OS.

registerDoParallel(cores = 2)
foreach (x in 1:100) %:%
    foreach (y in 1:100) %dopar% {
       readr::read_csv(paste0('fake_file',x,'_',y,'csv'))

       # work on files
       readr::write_csv(paste0('fake_file',x,'_',y,'csv'))
}
parallel-processing r
2 Answers

Here is another solution using the future package behind the scenes

library(tidyverse)

dir.create("example_data")

for (i in 1:10) {
  df <- as.data.frame(matrix(sample(1000),ncol = 10))
  write_csv(df,file = str_c("example_data/",i,"test.csv"))
  
}


# Create a function that does what you want for one file (try to test it)

file_function <- function(file_name){
  df_read <- read_csv(file_name, col_types = cols())
  
  df_processed <- df_read %>% 
    mutate(across(everything(),.fns = ~ .x * 10))
  
  df_processed %>%
    write_csv(file_name)
  
  str_c(file_name," processed")
  
}

# get file names

file_names <- list.files(path = "example_data/",full.names = T)

# furrr functional solution


library(furrr)

# plan execution

# multisession if windows

# multicore if linux or mac

set.seed(123)
plan(multisession,workers = availableCores())


file_names %>% future_map(file_function,.options = furrr_options(seed = TRUE))

# realease memory

plan(sequential)

# realease hd
unlink("example_data",recursive = T)
user_71999660

Just use vroom

library(vroom)

vroom("your_path")
user_71999660
Related