I'm confused with spark.cores.max defination
What I've got online is
spark.cores.max represents the total number of CPU cores for the whole application
My cluster is having 8 worker nodes and 1 master node. Where every node has 4 cores and 100 GB ram.
Does it mean total number of cores per worker node or in all 8 worker nodes ?
If I set spark.cores.max to 12 and cores/executor is 4, what does it mean ?
From Spark docs:
When running on a standalone deploy cluster or a Mesos cluster in "coarse-grained" sharing mode, the maximum amount of CPU cores to request for the application from across the cluster (not from each machine). If not set, the default will be spark.deploy.defaultCores on Spark's standalone cluster manager, or infinite (all available cores) on Mesos.
spark.cores.max limit the number of cores that spark job will use. If you do not set it will use 'spark.deploy.defaultCores' value.
In your example spark.executor.cores = 4 and spark.cores.max = 12 will indicate that your job will have three executors (12/4) of 4 cores each one.
set spark.cores.max to 12 and cores/executor is 4
This property will allow you to run several jobs in concurrently in the same spark cluster.
Regards