Using Spark 3.5 on AWS EMR 6.5.
I have 3 datasets.
Each Dataset has IDs, month, and a random number from 1 to 100 assigned to each ID.
Each Dataset is repartitioned by month, then written as partition by month, bucket by random number, sort by ID.
So each Dataset is confined to the same set of IDs and follows the same bucketing scheme.
When I go to join the three datasets on ID, month, and random number, I still see and Exchange in the SQL DAG.
Does anyone have any thoughts on why this would occur?