Why does AWS EMR EC2 instances have default spark.yarn.executor.memoryOverhead set to 18.75%

Viewed 196

Amazon EMR default memory limits for Spark executors
With the Spark on YARN configuration option which was introduced in EMR version 5.22: spark.yarn.executor.memoryOverheadFactor and defaults to 0.1875 (18.75% of the spark.yarn.executor.memoryOverhead setting )

https://ec2spotworkshops.com/running_spark_apps_with_emr_on_spot_instances/right_sizing_executors.html

spark.executor.memoryOverhead = executorMemory * 0.10, with minimum of 384
https://spark.apache.org/docs/latest/configuration.html

Can anyone explain why AWS EMR instances have 18.75% overhead as default but in spark documentation default is 10%?
Any particular reason as to why default to 18.75%??

0 Answers
Related