I am struggling to understand how OOM errors can happen in Apache Spark executors when Unified Memory Management is used.
From what I have read so far I conclude the following:
- Executor memory is divided in: execution memory, storage memory and user memory
- User memory is statically allocated = TOTAL_MEMORY * (1 - spark.memory.fraction).
User Memory. This is the memory pool that remains after the allocation of Spark Memory, and it is completely up to you to use it in a way you like. You can store your own data structures there that would be used in RDD transformations. For example, you can rewrite Spark aggregation by using mapPartitions transformation maintaining hash table for this aggregation to run, which would consume so called User Memory. [...] And again, this is the User Memory and its completely up to you what would be stored in this RAM and how, Spark makes completely no accounting on what you do there and whether you respect this boundary or not. Not respecting this boundary in your code might cause OOM error.
- The remaining executor memory is dynamically asigned between execution and storage as it gets requested. When all memory is in use and extra execution memory is needed, storage memory will be realeased by spilling it to disk and reasigned to execution memory. It can be configured a fraction of memory that won't be reassigned as execution memroy if it is used as storage memory (default 50%). In case there is no more memory can be allocated for either storage or execution pages will be spilled to disk.
So my question is, when I am executing pure Spark SQL queries, if execution memory and storage memory can be spilled to disk, how can OOM errors happen assuming user memory is not being used at all? Is it not supposed to take care of memory issues by spilling the data to disk? Are all OOM errors related to user memory fraction?
NOTE: I am not including OffHeap memory in the question on purpose to keep that complexity out of the problem