I've noticed that spark can sometimes schedule all tasks of a job onto the same executor when other executors are busy processing other job tasks
Here was a quick example on the spark shell, I created 3 executor with 1 core each
#1 sc.parallelize(List.range(0,1000), 2).mapPartitions(x => { Thread.sleep(100000); x }).count()
#2 sc.parallelize(List.range(0,1000), 10).mapPartitions(x => { Thread.sleep(1000); x }).count()
Job #1 hogs up the 2 executors by sleeping, and job #2 ends up running all of its 10 tasks sequentially on the 3rd executor. While it is understandable from sparks perspective that there weren't enough resources, but it should atleast try to distribute tasks even if it means they get delayed? What happens now is that the 1 executor becomes a hotspot for the next stage in that job because all the shuffle files have been persisted on that executor
While this is a manually generated example, we have noticed in our production setup where jobs with say 10k tasks are only being executed on a handful of executors
Just to debrief you a bit,
We create a single application and run several jobs ( 16 jobs in parallel sometimes even 48 ) within it. Should we have 1 application per job instead? What's the rationale behind creating 1 application vs multiple? All jobs are usually independent of each other