Need help to create loop in R that reads multiple files from AWS S3 bucket, appends them and saves them in SQLite table. I was appending each file individually when there were only 8 of them, but now there are over 1,000 and I need a loop to run through all the files. Here is the code I used to use when I had only 8:
require(aws.s3)
require(data.table)
require(tidyverse)
require(rjson)
require(parallel)
require(DBI)
# Set Up SQLite -----------------------------------------------------------
datamart <- dbConnect(RSQLite::SQLite(), "datamart.sqlite")
# RM s3 bucket credentials ------------------------------------------------
Sys.setenv(
"AWS_ACCESS_KEY_ID" = 'HIDDEN',
"AWS_SECRET_ACCESS_KEY" = 'HIDDEN',
"AWS_DEFAULT_REGION" = "us-east-1")
I would read the first file...
# Load policy files -------------------------------------------------------
tempfile <- tempfile()
save_object(object = "s3://thg_bucket/folder1/folder2/file.000", file = tempfile)
pol_temp <- read.csv(file = tempfile, sep = "|", quote = NULL) %>%
mutate(across(everything(), ~ map_chr(.x, ~ str_sub(string = .x, start = 2, end = -2))))
dbWriteTable(datamart, "policy", pol_temp, overwrite = T)
rm(pol_temp)
...and then append every other file, repeating the code below for file.001 to file.007
tempfile <- tempfile()
save_object(object = "s3://thg_bucket/folder1/folder2/file.001", file = tempfile)
pol_temp <- read.csv(file = tempfile, sep = "|", quote = NULL) %>%
mutate(across(everything(), ~ map_chr(.x, ~ str_sub(string = .x, start = 2, end = -2))))
dbWriteTable(datamart, "policy", pol_temp, append = T)
rm(pol_temp)
But, now there are 1,154 files to append in this way. I tried the following but it isn't working. Any correction to my code or an alternative approach is appreciated. I also need the first file processed to be overwrite and all those after to be append. Note 'Key' is the file name.
# Load policy files -------------------------------------------------------
path = "s3://thg_bucket/folder1/folder2/"
file.names <- data.table::rbindlist(get_bucket("thg_bucket", prefix = "folder1/folder2")) %>%
mutate(Key = str_sub(Key, 28, 37)) %>%
select(Key) %>% as.list()
# Read and Append Files
for(i in 1:length(file.names)){
tempfile <- tempfile()
save_object(object = paste0(path, file.names[i]), file = tempfile)
pol_temp <- read.csv(file = tempfile, sep = "|", quote = NULL) %>%
mutate(across(everything(), ~ map_chr(.x, ~ str_sub(string = .x, start = 2, end = -2))))
dbWriteTable(datamart, "policy", pol_temp, append = T)
rm(pol_temp)
}