I am new new to pyspark, i read somewhere "By applying bucketing on the convenient columns in the data frames before shuffle required operations, we might avoid multiple probable expensive shuffles. Bucketing boosts performance by already sorting and shuffling data before performing sort-merge joins"
so keen to know, how we can "avoid multiple probable expensive shuffles" with bucketing before join two dataframe
Here's a great article that helps you understand bucketing sorting.
Basically it comes down to you will "pre-chew" your data so that it's easy to join. You do this by using creating table definitions with CLUSTERED BY and BUCKET. If you regularly join two tables using identical clusterd by/bucketing on both tables will enable super fast joins between the tables. (They can be mapped to the same reducer to speed up the join).
