Add "filename" column to table as multiple files are read and bound

user_3699927 0
Viewed 13501

I have numerous csv files in multiple directories that I want to read into a R tribble or data.table. I use "list.files()" with the recursive argument set to TRUE to create a list of file names and paths, then use "lapply()" to read in multiple csv files, and then "bind_rows()" stick them all together: 

filenames <- list.files(path, full.names = TRUE, pattern = fileptrn, recursive = TRUE)
tbl <- lapply(filenames, read_csv) %>% 
  bind_rows()

This approach works fine. However, I need to extract a substring from the each file name and add it as a column to the final table. I can get the substring I need with "str_extract()" like this:

sites <- str_extract(filenames, "[A-Z]{2}-[A-Za-z0-9]{3}")

I am stuck however on how to add the extracted substring as a column as lapply() runs through read_csv() for each file.

lapply r
6 Answers

tidyverse approach:

Update:

readr 2.0 (and beyond) now has built-in support for reading a list of files with the same columns into one output table in a single command. Just pass the filenames to be read in the same vector to the reading function. For example reading in csv files:

(files <- fs::dir_ls("D:/data", glob="*.csv"))
dat <- read_csv(files, id="path")

Alternatively using map_dfr with purrr: Add the filename using the .id = "source" argument in purrr::map_dfr() An example loading .csv files:

 # specify the directory, then read a list of files
  data_dir <- here("file/path")
  data_list <- fs::dir_ls(data_dir, regexp = ".csv$")

 # return a single data frame w/ purrr:map_dfr 
 my_data = data_list %>% 
    purrr::map_dfr(read_csv, .id = "source")
  
 # Alternatively, rename source from the file path to the file name
  my_data = data_list %>% 
    purrr::map_dfr(read_csv, .id = "source") %>% 
    dplyr::mutate(source = stringr::str_replace(source, "file/path", ""))
user_19733690

data.table approach:

If you name the list, then you can use this name to add to the data.table when binding the list together.

workflow

files <- list.files( whatever... )
#read the files from the list
l <- lapply( files, fread )
#names the list using the basename from `l`
# this also is the step to manipuly the filesnamaes to whatever you like
names(l) <- basename( l )
#bind the rows from the list togetgher, putting the filenames into the colum "id"
dt <- rbindlist( dt.list, idcol = "id" )
user_63562780
Related