While using these line of code which is in the documenttion of Pyspark unionByName
df =spark.createDataFrame(data,columns)
df1 = spark.createDataFrame([[1, 2, 3]], ["col0", "col1", "col2"])
df2 = spark.createDataFrame([[4, 5, 6]], ["col1", "col2", "col3"])
df1.unionByName(df2, allowMissingColumns=True).show()
got this error:
unionByName() got an unexpected keyword argument 'allowMissingColumns'
Traceback (most recent call last):
TypeError: unionByName() got an unexpected keyword argument 'allowMissingColumns'
Is there any other way to do this, in my code
df1: ["col0", "col1", "col2","col3"]
df2: ["col1", "col2"]
I want to union them on ["col1", "col2"] and expect the col0 and col3 to be null after union.