Add columns to dataframe that are not already in another dataframe

user_7331016 0
Viewed 150

I am trying to add empty columns to a dataframe df1 that are not already in a second dataframe df2. So, given

df2.columns = ['a', 'b', 'c', 'd']
df1.columns = ['a', 'b']

I would like to add columns with names 'c' and 'd' to dataframe df1.

For performance reasons, I would like to avoid a loop with multiple withColumn() statements:

for col in df1.columns:
    if col not in df2.columns:
        df1= df1.withColumn(col, lit(None).cast(StringType()))

My first attemt

df1 = df1.select(col('*'),
                 lit(None).alias(col_name) for col_name in df1.columns if col_name not in df2.columns)

is throwing an error

TypeError: Invalid argument, not a string or column: <generator object myfunction.. at 0x7f60e2bcc8e0> of type <class 'generator'>. For column literals, use 'lit', 'array', 'struct' or 'create_map' function.

pyspark apache-spark-sql apache-spark python
1 Answers

You need first to convert generator to list using list() function. After converting pass the list to select().

df1.select(col('*'), *list(lit(None).alias(col_name) for col_name in df2.columns if col_name not in df1.columns))
user_87733090
Related