how to merge 2 or more dataframes with pyspark

user_14036099 0
Viewed 40

i have 3 pyspark df as and i need to single df by merging all df ,as

+--------+
|        |
|  name  |
|        |
|--------|    
| orange |  
+--------+

+--------+
|        |
|  age   |
|        |
|--------|    
|  10    |  
+--------+

+---------+
|         |
|  place  |
|         |
|---------|    
|  delhi  |  
+---------+

and the output df should be like below

+---------+---------+---------+
|         |         |         |
|  name   |  age    |  place  |
|         |         |         |
|---------+---------+---------+    
|         |         |         |
| orange  |  10     |  delhi  |
|         |         |         |
+---------+---------+---------+

Does anyone know a solution?

thanks in advance

pyspark dataframe
1 Answers

Use either crossJoin,row_number for this case

Example:

df=spark.createDataFrame([('orange',)],['name'])
df1=spark.createDataFrame([(10,)],['age'])
df2=spark.createDataFrame([('delhi',)],['place'])

df.crossJoin(df1).crossJoin(df2).show()
#+------+---+-----+
#|  name|age|place|
#+------+---+-----+
#|orange| 10|delhi|
#+------+---+-----+

#using window
from pyspark.sql import *
from pyspark.sql.functions import *
w=Window.orderBy(lit(1))
df=df.withColumn("rn",row_number().over(w))
df1=df1.withColumn("rn",row_number().over(w))
df2=df2.withColumn("rn",row_number().over(w))
df.join(df1,['rn'],'inner').join(df2,['rn'],'inner').drop('rn').show()
#+------+---+-----+
#|  name|age|place|
#+------+---+-----+
#|orange| 10|delhi|
#+------+---+-----+
user_76326950
Related