I've two medium sized tables (tableA and tableB) where I'm joining tableB twice with tableA on the same join key.
...
from tableA A
left join tableB B1 on A.col1 = B1.col1
left join tableB B2 on A.col2 = B2.col1
I'm seeing that tableB is getting exchanged twice in the spark DAG.
Since, the exchange is happening on the same join key (col1), I was curious on whether the 2nd exchange can be avoided?
Yes, in principle this is possible and there are two techniques you can consider:
caching - you can repartition the table B and then cache it and use this cached table in your joins. That will make sure that it will be shuffled only once. It is however not very suitable for large tables that will not fit in the memory.
reused exchange - if the Exchanges are identical, Spark will reuse the Exchange automatically. In your case, I can see from the plan that they are not identical because the second Exchange is showing more data (419MB in the second Exchange vs 325MB in the first Exchange). This might be caused for example by different projection (maybe you select different columns in each case later on and the columnPruning rule will prune the columns). If you can make sure that both Exchanges are identical (the sub-branch that is before the Exchange operator has the same operators with the same expressions as the second Exchange sub-branch) Spark will reuse it and you will see in the plan the ReusedExchange operator.
You may check my recent article about the technique of reusing the Exchange.