How can I write dataframe to csv file using one partition although the file size exceeds executors memory

user_15070559 0
Viewed 325

I am working on Apache Spark standalone cluster with 2 executors, each having 1g heap space and 8 cores each.

I load input file having size 2.7Gb into a dataframe df. This was successfully done using 21 tasks, that is I used 21 partitions in total across my whole cluster.

Now I tried writing this out to csv using only 1 partition, so that I get all my records in 1 csv file.

df.coalesce(1).write.option("header","true").csv("output.csv")

I expected to get an OOM error since the total usable memory for an executor is less than 2.7Gb. But this did not happen.

How did my task not break despite the data being larger than a single partition? What exactly is happening here under the hood?

apache-spark-sql apache-spark hadoop jvm scala
1 Answers

The original csv file is of size 2.7GB in its raw format (text-based, no compression). When you read that file with Spark it splits up the data into multiple partitions based on the configuration spark.files.maxPartitionBytes which defaults to 128MB. Doing the math leads to 2700MB / 128MB = 21 partitions.

Spark keeps the data in-memory but in its own storage format which is called "Vectorized Parquet" and using a default compression "lz4".

Therefore, the 2.7GB will fit into the provided 1GB memory.

Keep in mind, that not all 100% of the 1GB is available to use for data store/processing. There is a clear design to the executors memory that can be configured by the configuration spark.memory.fraction and spark.memory.storageFraction. I have written an article on medium about the Executor Memory Layout.

Here is a picture that helps to understand the Memory Layout:

enter image description here

user_122089100
Related