combine multiple text files into one single tsv file

user_15881499 0
Viewed 463

I have 254 text files in total. I want to convert all the text file into one single tsv file in r-programming. How can I do it? Even if it's not possible in r, is there any other way I can do it?

 path <- setwd("~/syeda_demo/transcripts/")

 files <- list.files(path = 'path', 
                pattern = '\\.txt$', full.names = TRUE)

map_dfr(files, read_table) %>%
      write_tsv('path/newfile.tsv')`
data-analysis r database
2 Answers

We get the list of files from the folder with list.files, then loop over the files, use read_table from readr to read all of them, row bind them into a single data set (_dfr) and write it back as new file with write_tsv

library(readr)
library(purrr)
library(dplyr)
files <- list.files(path = '/path/to/your/folder', 
            pattern = '\\.txt$', full.names = TRUE)
map_dfr(files, read_table) %>%
      write_tsv(path = '/path/to/your/folder/yourfile.tsv')

If the files are really big, we could use fread/fwrite from data.table

library(data.table)
fwrite(rbindlist(lapply(files, fread), fill = TRUE), 
    file = "/path/to/your/folder/yourfile.tsv", sep="\t")
user_37322710

It is much more memory efficient to do this in a shell system command. For example on Mac OS or Linux enter the following in the Terminal:

cat ~/syeda_demo/transcripts/*.txt > ~/syeda_demo/transcripts/path/newfile.tsv

This will print all text files into a single file so that every line in each file will be a new row in your tsv.

You can run this within an R session with the "system" command:

> system("cat ~/syeda_demo/transcripts/*.txt > ~/syeda_demo/transcripts/path/newfile.tsv")

To read such a large file into R I recommend data.table::fread or vroom::vroom.

user_42166250
Related