I am running spark jobs on GKE Autopilot (Google's kubernetes engine). My workload is not very CPU intensive so CPU is idle a lot.
In spark-submit, I use spark.executor.cores and spark.executor.instances and GKE provisions appropriately sized nodes and starts executor pods based on this config.
However, there is 1:1 correlation between kubernetes pod core and spark task core. So, spark parallelism is limited to spark.executor.cores * spark.executor.instances. This causes very low CPU utilization and thus high cost of running.
In spark local mode, I can give more cores than are physically present and it is upto the OS to schedule and run the threads in parallel. In spark standalone mode, there is SPARK_WORKER_CORES which can also overcommit CPU.
There is spark.task.cpus which does the opposite of what I want.
Is there a way to overcommit cores when running in kubernetes?