Our Spark clusters have autoscaling enabled that can go from 3 to 60 nodes.
The current guidance is set spark.default.parallelism & spark.sql.shuffle.partitions at the upper limit of autoscaling. For example 60 nodes x 8 vCPUs x 3 = 1,440.
However, that seems to be a lot of data partitions for 3 slave nodes to start.
It appears setting spark.default.parallelism & spark.sql.shuffle.partitions dynamically would be optimal. Is there guidance on whether we should or should not dynamically set this?
And if we can set spark.default.parallelism & spark.sql.shuffle.partitions dynamically, how can it be done, via spark-submit?