Adaptive Query Execution in Spark 3

user_6257282 0
Viewed 1006

I have just learned about the new Adaptative Query Execution (AQE) introduced with Spark 3.0.

However there is something that I feel weird. For the following example of switching join strategy: enter image description here

The stages 1 and 2 had completely finished (including the map side shuffle) before the AQE decided to switch to the broadcast mode.

My questions: Isn't that too late to switch to broadcast since the two dataset have been already written to disk to be shuffled (map side shuffle)? Will this switch be in most cases more efficient than continuing with the reduce side shuffle? I guess yes since Databricks guys have done this choice but I would like to make sure I didn't miss something..

apache-spark-sql apache-spark shuffle optimization performance
1 Answers

Isn't that too late to switch to broadcast since the two dataset have been already written to disk to be shuffled (map side shuffle)? - Totally valid concern, but "better late than never" right? ;-) Spark Performance Tuning mentions:

...This is not as efficient as planning a broadcast hash join in the first place, but it’s better than keep doing the sort-merge join, as we can save the sorting of both the join sides, and read shuffle files locally to save network traffic (if spark.sql.adaptive.localShuffleReader.enabled is true)

spark.sql.adaptive.localShuffleReader.enabled runtime config was added in Spark 3.0 and is set to true by default.

I also think it is something that can help / be built upon once executor-side broadcast SPARK-17556 comes about.

user_6387640
Related