generating join condition dynamically in pyspark

user_13559272 0
Viewed 188

Can someone suggest a way to pass a listofJoinColumns and a condition to joins in pyspark.

e.g. I need the columns to be joined on to be dynamically taken from a list and also want to pass another condition on the join. Something similar to this done in scala is explained here: generating join condition dynamically in spark/scala

I am looking for a similar solution in pyspark.

I understand that I can use the join e.g. a.join(b , listofjoincolumns, how="inner") but I want to pass a join condition as well:

I want to call it as a.join(b , listofjoincolumns and join condition, how="inner")

Can someone please suggest a way to do so in pyspark.

pyspark dynamic join
1 Answers

Try to convert the list of join columns to a join condition itself:

from functools import reduce
from operator import and_
df_a.join(df_b, reduce(and_,
                       [df_a[col] == df_b[col] for col in listofcols],
                       joinCond)
         )
user_141657300
Related