I have 20 yarn/hdfs nodes.
When I submit a Spark job to Yarn, Yarn start containers on the 1st node until it's full, then moves to the 2nd, the 3rd until all executors are created.
So I endup with 3 nodes out of 20 running heavy.
I tried all these, and nothing seems to help. --conf spark.scheduler.mode=FAIR --conf spark.sql.sources.ignoreDataLocality.enabled=true --conf spark.locality.wait=0 --conf spark.locality.wait.node=0.
Running Spark 3.1 and Yarn 3.1.1
Any ideas?