Spark "Task serialization failed: java.lang.OutOfMemoryError"

Viewed 131

I try write in parquet files 300 rows each contain column BinaryType with length ~ 14 000 000. And as result i got this Exception:

org.apache.spark.SparkException: Job aborted due to stage failure: Task serialization failed: java.lang.OutOfMemoryError java.lang.OutOfMemoryError at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)

Bur this work when i try write 100 and 200 rows with same value.

spark submit configuration:

            <param>--driver-memory 20G</param>
            <param>--num-executors 100</param>
            <param>--executor-memory 20G</param>
            <param>--executor-cores 3</param>
            <param>--driver-cores 3</param>
            <param>--conf spark.driver.maxResultSize=10G</param>
            <param>--conf spark.executor.memoryOverhead=10G</param>
            <param>--conf spark.driver.memoryOverhead=10G</param>
            <param>--conf spark.network.timeout=1200000</param>
            <param>--conf spark.memory.offHeap.enabled=true</param>
            <param>--conf spark.sql.files.maxRecordsPerFile=2</param>

I really can't understand how 14 mb per row can throw an OutOfMemoryError when trying to write. If i ran show on the final DataFrame, it works without errors.

used persist(StorageLevel.MEMORY_AND_DISK_SER) at final DataFrame also didnt help

0 Answers
Related