Let's say I have the following config:
Instance type: r5.8xlarge (32 vCore, 256 GiB memory, EBS only storage)
EBS Storage: 100GiB
No. of on demand units: 3995
Per Instance counts = 4 unit
No. of EC2 instances = 3995/4 = 999 (But EMR provisioned 1004 instances)
spark.executor.memoryOverhead=10240mb
spark.driver.memory=204800mb
spark.executor.memory=51200mb
--executor-cores 7
spark.memory.fraction=0.9
From the EMR console, I can see the total number of containers allocated is 826, but this doesn't align with my understanding of the calculation.
Here is my calculation based on above config:
No of executors per instance = Instance memory * spark_memory_fraction/(Executor memory + its overhead) = 256GB*0.9/(51+10) = 3.7 executors/instance ~ Math.floor(3.7) = 3 executor/instance
Overall executors = 999 instances * 3 executor/instance = 2997 executors.
The number that I see in the console(826 containers or executors) vastly differs from my calculation(2997 executors). Can someone help me understand this how its calculated and why there is so much difference ?