How Spark achieves memory fairness between tasks?

Question

I was watching this presentation that is about spark memory management.

He talks about how they achieve fairness between different tasks in one executor (12:00). And he presents the idea of dynamic assignment between tasks and he declares that Spark spill other task's pages to disk if more tasks begin to execute.

I read before tasks in Spark are essentially threads and in Java we don't have this capability to manage the memory of threads and establish memory fairness between them. I wonder how Spark achievs this?

user_6933993 · Answer

I am not sure about fairness being valid.

Here is an excellent read on executor memory divided over more than 1 core per Executor https://www.davidmcginnis.net/post/spark-job-optimization-myth-5-increasing-executor-cores-is-always-a-good-idea, which has nothing to do with spilling to disk.
Spilling to disk is if too large for available memory. Also a partition can be completely evicted, resulting in re-computation (from last checkpoint).

Where I have been the paradigm was similar to this article in that we ran more Executors with a single Core. The default from Spark is indeed 1 as well, I would assume that that was a thought out default from the vendor.

How Spark achieves memory fairness between tasks?

1 Answers