Facing large data spills for small datasets on spark

user_466521 0
Viewed 430

I am trying to run some spark sql on NOA datasets available here:

https://www.ncei.noaa.gov/data/global-summary-of-the-day/access/2021/

I am trying to run some query which involves grouping and sorting.

df
      .groupBy("COUNTRY_FULL")
      .agg(max("rank"), last("consecutive").as("consecutive"))
      .withColumn("maxDays", maxDaysTornodoUdf(col("consecutive")))
      .sort(col("maxDays").desc)
      .limit(1)
      .show()

The input size is just 50 MB zipped csvs and I am running this locally (4 cores). These are the settings I use.

  • spark.driver.memory: 14g
  • spark.sql.windowExec.buffer.in.memory.threshold: 20000
  • spark.sql.windowExec.buffer.spill.threshold : 20000
  • spark.sql.shuffle.partitions : 400

I see too many disk spills for such a small data

21/08/16 10:23:13 INFO UnsafeExternalSorter: Thread 54 spilling sort data of 416.0 MB to disk (371  times so far)
21/08/16 10:23:14 INFO UnsafeExternalSorter: Thread 79 spilling sort data of 416.0 MB to disk (130  times so far)
21/08/16 10:23:14 INFO UnsafeExternalSorter: Thread 53 spilling sort data of 400.0 MB to disk (240  times so far)
21/08/16 10:23:14 INFO UnsafeExternalSorter: Thread 69 spilling sort data of 400.0 MB to disk (24  times so far)
21/08/16 10:23:16 INFO UnsafeExternalSorter: Thread 54 spilling sort data of 416.0 MB to disk (372  times so far)
21/08/16 10:23:16 INFO UnsafeExternalSorter: Thread 79 spilling sort data of 416.0 MB to disk (131  times so far)

However, when I check the Spark UI, the spillage doesnt seem to be much

enter image description here

Eventually the spark job terminates with error Not Enough memory I do not understand what is happening.

apache-spark-sql apache-spark scala
1 Answers

You are using 400 as spark.sql.shuffle.partitions, which is too much for the data size which you are dealing with.

Having more shuffle partitions for lesser amount of data causes more partitions/tasks and it will reduce the performance. Read best practices to configure shuffle partition here.

Try reducing shuffle partitions. You may try setting it to spark.sparkContext.defaultParallelism.

user_87733090
Related