Let's say I have multiple DataFrames partitioned by same column.
When I now try to do operations (like joins, creating objects of them and using in a logic, etc.) is it ensured that all the data concerned to a particular worker node is already present there (while partitioning)?
Otherwise it will cause shuffle which is expensive. Ex.
If DFs are as follows:
d1, d2 => data in columns
p1, p2 => Partition
DF1 => (d1, d2, d3, p1)
(d4, d5, d6, p2)
DF2 => (d7, d8, d9, p1)
(d10, d11, d12, p2)
Then will the partitioning be as follows, i.e. Worker nodes having the required data locally?
Workernode 1: (d1, d2, d3, p1)
(d7, d8, d9, p1)
Workernode 2: (d4, d5, d6, p2)
(d10, d11, d12, p2)