I am trying to understand how Adaptive query execution and spark.sql.shuffle.partitions interact in spark 2.4 (though if this changed in spark 3.0 that would be interesting to note as well).
If I set AQE to true (unlike spark 3.0 it is False by default in spark 2.4), could it choose higher and lower number of partitions? Or does it depend on whether I set spark.sql.adaptive.coalescePartitions.enabled to true?
In my rather large application, my code used to crash until I specified enough partitions. AQE was enabled but somehow wasn't able to do that for me. After setting the partitions explicitly (With AQE still on), the code works reliably.
(I am using pyspark, spark 2.4 on CDH)