Amazon EMR default memory limits for Spark executors
With the Spark on YARN configuration option which was introduced in EMR version 5.22: spark.yarn.executor.memoryOverheadFactor and defaults to 0.1875 (18.75% of the spark.yarn.executor.memoryOverhead setting )
spark.executor.memoryOverhead = executorMemory * 0.10, with minimum of 384
https://spark.apache.org/docs/latest/configuration.html
Can anyone explain why AWS EMR instances have 18.75% overhead as default but in spark documentation default is 10%?
Any particular reason as to why default to 18.75%??