PySpark - how to update Dataframe by using join?

user_5285068 0
Viewed 3083

I have a dataframe a:

id,value
1,11
2,22
3,33

And another dataframe b:

id,value
1,123
3,345

I want to update dataframe a with all matching values from b (based on column 'id').

Final dataframe 'c' would be:

id,value
1,123
2,22
3,345

How to achieve that using datafame joins (or other approach)?

Tried:

a.join(b, a.id == b.id, "inner").drop(a.value)

Gives (not desired output):

+---+---+-----+
| id| id|value|
+---+---+-----+
|  1|  1|  123|
|  3|  3|  345|
+---+---+-----+

Thanks.

pyspark apache-spark sql
1 Answers

I don't think there is an update functionality. But this should work:

import pyspark.sql.functions as F

df1.join(df2, df1.id == df2.id, "left_outer") \
   .select(df1.id, df2.id, F.when(df2.value.isNull(), df1.value).otherwise(df2.value).alias("value")))
user_46589730
Related