Writing data.table's to Spark with sparklyr efficiently

user_862288 0
Viewed 622

I'm trying to upload large data.table's into Spark using sparklyr as follows:

copy_to(sc, obj, "name_of_object")

where sc is a Spark connection and obj my object. After some digging, I found that writing tables to Spark is done as follows (in spark_data_copy()): a csv file is created on disk using the serializer spark_serialize_csv_scala which uses base R write.table(). After writing the file, R instructs Spark to read the csv.

I found this to be not very efficient for my use R seems to hang and there is only so much tuning one could do on the csv writing part of the task. Does an efficient way exist to put data already in R memory into Spark?

sparklyr apache-spark r
1 Answers

One solution is to use data.table's fwrite() to write a csv and import that using spark_read_csv(). This gives two speed ups:

  1. fwrite is a lot faster than write.table()
  2. For some reason, also using spark_read_csv() is a lot faster, run like this the local Spark instance will use all cores (and maybe there are more differences). For me, on a 16 core machine that really helps.

If given time, I will make more precise measurements.

user_8622880
Related