Parsing 150Kbyte text is very slow in R

user_266068 0
Viewed 51

I would like to extract some data from a small txt file.

I also need to apply some regular expressions on every line to match the contents inside brackets on every line. For 5000 lines the code is extremely slow.

I use the following code:

file_list <- list.files(pattern = "^q.*.txt")

start_time <- Sys.time()
for (file in file_list){
  res <- length(readLines(file))
  for (i in 1:res) {
    line = readLines(file, n = i)
    brackets <- unlist(regmatches(line, gregexpr("\\[[^][]*]",line)))
    brackets <- gsub("[", "", brackets, fixed = TRUE)
    brackets <- gsub("]", "", brackets, fixed = TRUE)
    # df <- colsplit(brackets, pattern = ",",names=c("S1", "S2", "S3", "S4"))
  }
}

end_time <- Sys.time()

total <- end_time - start_time
print(total)

Here is the code and the file: https://repl.it/repls/SkinnySiennaWebsites

How can I parse sufficiently large files faster?

r parsing
1 Answers

Your inner for loop is unnecessary -- all of the string manipulation functions you're using are vectorized, so you can simply do:

for (file in file_list) {
  lines <- readLines(file)
  brackets2 <- unlist(regmatches(lines, gregexpr("\\[[^][]*]", lines)))
  brackets2 <- gsub("[", "", brackets, fixed = TRUE)
  brackets2 <- gsub("]", "", brackets, fixed = TRUE)
}

identical(brackets, brackets2)
#> TRUE
user_94308550
Related