Use R loop to load and append multiple files from AWS S3 bucket and save them as SQLite table

Viewed 22

Need help to create loop in R that reads multiple files from AWS S3 bucket, appends them and saves them in SQLite table. I was appending each file individually when there were only 8 of them, but now there are over 1,000 and I need a loop to run through all the files. Here is the code I used to use when I had only 8:

require(aws.s3)
require(data.table)
require(tidyverse)
require(rjson)
require(parallel)
require(DBI)

# Set Up SQLite -----------------------------------------------------------
datamart <- dbConnect(RSQLite::SQLite(), "datamart.sqlite")

# RM s3 bucket credentials ------------------------------------------------
Sys.setenv(
  "AWS_ACCESS_KEY_ID" = 'HIDDEN',
  "AWS_SECRET_ACCESS_KEY" = 'HIDDEN',
  "AWS_DEFAULT_REGION" = "us-east-1")

I would read the first file...

# Load policy files -------------------------------------------------------
tempfile <- tempfile()
save_object(object = "s3://thg_bucket/folder1/folder2/file.000", file = tempfile)
pol_temp <- read.csv(file = tempfile, sep = "|", quote = NULL) %>%
  mutate(across(everything(), ~ map_chr(.x, ~ str_sub(string = .x, start = 2, end = -2))))

dbWriteTable(datamart, "policy", pol_temp, overwrite = T)
rm(pol_temp)

...and then append every other file, repeating the code below for file.001 to file.007

tempfile <- tempfile()
save_object(object = "s3://thg_bucket/folder1/folder2/file.001", file = tempfile)
pol_temp <- read.csv(file = tempfile, sep = "|", quote = NULL) %>%
  mutate(across(everything(), ~ map_chr(.x, ~ str_sub(string = .x, start = 2, end = -2))))

dbWriteTable(datamart, "policy", pol_temp, append = T)
rm(pol_temp)

But, now there are 1,154 files to append in this way. I tried the following but it isn't working. Any correction to my code or an alternative approach is appreciated. I also need the first file processed to be overwrite and all those after to be append. Note 'Key' is the file name.

# Load policy files -------------------------------------------------------
path = "s3://thg_bucket/folder1/folder2/"
file.names <- data.table::rbindlist(get_bucket("thg_bucket", prefix = "folder1/folder2")) %>% 
  mutate(Key = str_sub(Key, 28, 37)) %>% 
  select(Key) %>% as.list()

# Read and Append Files
for(i in 1:length(file.names)){
  tempfile <- tempfile()
  save_object(object = paste0(path, file.names[i]), file = tempfile)
  pol_temp <- read.csv(file = tempfile, sep = "|", quote = NULL) %>%
    mutate(across(everything(), ~ map_chr(.x, ~ str_sub(string = .x, start = 2, end = -2))))
  
  dbWriteTable(datamart, "policy", pol_temp, append = T)
  rm(pol_temp)  
}
0 Answers
Related