Databricks uses a "driver daemon" is a proprietary wrapper around the underlying implementation of OSS Spark. This daemon is opaque and difficult to troubleshoot.
I'm attempting to use an all purpose cluster to perform a series of independent "jar tasks" in parallel. I allow a number of concurrent tasks to run at a time (10 or more) and they are not intended to be synchronized or serialized in any way. They are independent.
However as I scale up the workload, I find that I hit a bottleneck on single-threaded CPU activity within the driver. The limit that is reached is well below max CPU available on the driver node. It appears to be about ~25% of the CPU.
See the screenshot below. It contains the CPU usage of the driver node. Regardless of whether I attempt to run 20, 10, or 30 concurrent jobs (respectively), the CPU won't grow above ~25%.
If anyone understands how a databricks "driver daemon" works, can you please help me with one primary question:
What is the resource that is being synchronized/serialized in the driver daemon? I'm assuming there is a resource bottleneck (critical section) rather than a deliberate attempt to serialize all the submitted jobs.
I realize that Databricks is supposed to be a high performing Spark platform. In light of that, this bottleneck is surprising to me.
Any help would be greatly appreciated.