Pyspark: TypeError: unionByName() got an unexpected keyword argument 'allowMissingColumns'

Viewed 1454

While using these line of code which is in the documenttion of Pyspark unionByName

df =spark.createDataFrame(data,columns)

df1 = spark.createDataFrame([[1, 2, 3]], ["col0", "col1", "col2"])
df2 = spark.createDataFrame([[4, 5, 6]], ["col1", "col2", "col3"])
df1.unionByName(df2, allowMissingColumns=True).show()

got this error:

    unionByName() got an unexpected keyword argument 'allowMissingColumns'
    Traceback (most recent call last):
    TypeError: unionByName() got an unexpected keyword argument 'allowMissingColumns'

Is there any other way to do this, in my code

   df1: ["col0", "col1", "col2","col3"]
   df2: ["col1", "col2"]

I want to union them on ["col1", "col2"] and expect the col0 and col3 to be null after union.

0 Answers
Related