Concatenate two DataFrames via column [PySpark]

user_5807433 0
Viewed 2462

I have two columns, i.e. (both with the same number of entries per column)

df1 =
+-------+
| col1  |
+-------+
|   10  |
+-------+
|   3   |
+-------+
...
df2 = 
+-------+
| col2  |
+-------+
|   6   |
+-------+
|   1   |
+-------+
...

I wish to merge them such that the final DataFrame is of the following shape:

df3 =
+-------+-------+
| col1  | col2  |
+-------+-------+
| 10    | 6     |
+-------+-------+
| 3     | 1     |
+-------+-------+
...

But I am not able to do so with the join method since I am not attempting to merge columns based on column header. If anybody has any tips on how to achieve this easily that would be greatly helpful!

pyspark dataframe concatenation merge
2 Answers

One way to do this if you are able to get your columns as list is to use python zip method. For example:

list1 = [1,2,3]
list2 = ['foo','baz','bar']
data_tuples = list(zip(list1,list2))
df = spark.createDataFrame(data_tuples)
df.show()

+---+---+
| _1| _2|
+---+---+
|  1|foo|
|  2|baz|
|  3|bar|
+---+---+

However I'm not sure how it behave with big dataset.

user_62734960

try this

df1 = df1.withColumn("code", monotonically_increasing_id())

df2 = df2.withColumn("code", monotonically_increasing_id())

this way you give them both a column code which you can use to classically merge both df's with.

df3 = df2.join(df1, ["code"])

user_107075610
Related