Fast delimited JSON writing with arrow?

user_7220020 0
Viewed 156

Is it possible to use the R arrow package to write JSON-LD? I know there are very fast methods to write CSV (arrow::write_csv_arrow) and parquet, and I have been using arrow::read_json_arrow for super fast reading of very large delimited JSON.

But to write I am still using writeLines which seems fairly slow. I wonder if there are plans to support this in arrow itself?

Thanks

Updated with reproducible example showing my current code. In reality the data is much larger and the JSON generation takes over 1 hour:

library(data.table)
library(arrow)
library(rbenchmark)

dt <- data.table(id = 1:1000)
for (i in seq(3)) dt[[paste0("numeric",i)]] <- runif(nrow(dt))
for (i in seq(4)) dt[[paste0("symbolic",i)]] <- as.character(paste0("X",round(1000*runif(nrow(dt)))))

listToJson <- function(l) {
  jsonlite::toJSON(l, auto_unbox = T, na = "null", digits = NA)
}

b <- benchmark( "row-wise json" = {
  writeLines(sapply(seq(nrow(dt)), function(r) { listToJson(as.list(dt[r,])) }), "test.json")
},
"write.csv" = {
  write.csv(dt, "test.csv")
},
"arrow::write_csv_arrow" = {
  arrow::write_csv_arrow(dt, "test.csv")
},
replications = 10
)

Results show how much slower my JSON writing is compared to CSV, even with write.csv:

test replications elapsed relative user.self sys.self user.child sys.child
arrow::write_csv_arrow 10 0.029 1.000 0.027 0.004 0 0
write.csv 10 0.054 1.862 0.051 0.004 0 0
row-wise json 10 5.056 174.345 5.014 0.040 0 0
apache-arrow r
1 Answers

It is not currently possible to do this with the R package (or the underlying C++ implementation). There is a ticket open on the project which mentions this, but it has not been worked on. I have added a comment on there highlighting that it would be useful to have.

In the meantime, perhaps if you tell us a bit more about your code and data (how many rows/columns etc, the code you're using to write it), we could see if there's anything we can do to help make it it more efficient.

user_11410500
Related