df_1 =
metadata_id col_1 col_2 col_3
100 1 2 3
101 1 2 1
102 2 5 3
103 1 2 6
104 1 4 2
105 6 2 3
df_2 =
my_metadata_id col_4 col_5 version
100 1 2 10
101 1 7 11
201 2 2 11
102 1 2 12
202 1 2 12
302 6 5 12
103 1 2 13
104 2 1 14
105 1 2 15
Desired output:
metadata_id col_1 col_2 col_3 version
100 1 2 3 10
101 1 2 1 11
102 2 5 3 12
103 1 2 6 13
104 1 4 2 14
105 6 2 3 15
It looks trivial, but when I do:
df_1 = df_1.join(df_2.select("version"), df_1("metadata_id") == df_2("my_metadata_id"), how="left")
I get: TypeError: 'DataFrame' object is not callable.
I would just join normally and then drop the extra columns, but in reality there is a lot of them and I wouldn't do it manually when there must be an elegant solution...