Currently, I am running Spark job through EMR and working on Spark tuning. I read about number of executor per instance and memory calculation etc, and I got confused based on the current setup.
So currently it uses spark.dynamicAllocation.enabled as true by default from EMR and spark.executor.cores as 4 (not set by me, I assume it is by default). Also use one r6.xlarge (32 GiB of Memory, 4 vCPUs) for master and two for cores.
In this case, based on the formula: Number of executors per instance = (total number of virtual cores per instance - 1)/ spark.executors.cores , (4 - 1) / 4 = 0. Would it be correct?
When I check spark UI, it added many executors. What information did I miss from this..?