Spark - StorageLevel (DISK_ONLY vs MEMORY_AND_DISK) and Out of memory Java heap space

user_2807379 0
Viewed 6502

Lately I've been running a memory-heavy spark job and started to wonder about storage levels of spark. I persisted one of my RDDs as it was used twice using StorageLevel.MEMORY_AND_DISK. I was getting OOM Java heap space during the job. Then, when I removed the persist completely, the job has managed to go through and finish.

I always thought that the MEMORY_AND_DISK is basically a fully safe option - if you run out of memory, it spills the object to disk, done. But now it seemed that it did not really work in the way I expected it to.

This derives two questions:

  1. If MEMORY_AND_DISK spills the objects to disk when executor goes out of memory, does it ever make sense to use DISK_ONLY mode (except some very specific configurations like spark.memory.storageFraction=0)?
  2. If MEMORY_AND_DISK spills the objects to disk when executor goes out of memory, how could I fix the problem with OOM by removing the caching? Did I miss something and the problem was actually elsewhere?
rdd apache-spark memory caching scala
2 Answers

So, after few years ;) that's what I believe happened:

  • Caching is not a way to save execution memory. The best you can do is not to lose execution memory (DISK_ONLY) when caching.
  • It's most likely the lack of execution memory that caused my job to throw OOM error, although I don't remember the actual use case.
  • I used MEMORY_AND_DISK caching and the MEMORY part took its part from the unified region which made it impossible for my job to finish (since the Execution = Unified - Storage memory was not enough to perform the job)
  • Due to above, when I removed caching at all, it took slower, but the job had enough execution memory to finish.
  • With DISK_ONLY caching it seems that the job would therefore finish as well (although not necessarily faster).

https://spark.apache.org/docs/latest/tuning.html#memory-management-overview

user_28073790
user_51889550
Related