How to pick number of executors , cores for each executor and executor memory

Question

10 Node cluster, each machine has 16 cores and 126.04 GB of RAM

Application input dataset is around 1TB with 10-15 files and there is some aggregation(groupBy)

Job will run using Yarn as resource scheduler

My Question how to pick num-executors, executor-memory, executor-core, driver-memory, driver-cores?

user_9108912 · Answer

Additionally you can understand how Spark Memory works - https://luminousmen.com/post/dive-into-spark-memory