How to remove column duplication in PySpark DataFrame without declare column name where the columns are unknown to me?

Viewed 13

I have one source data:

employee_id employee_name salary designation
101 monali 70000 developer 102 Amy 70000 developer 103 neha 65000 tester

I have a schema : ["employee_id","employee_name","salary","designation" ]

Now I got 1 extra column in the source data which is unknown to me, so it is not in the schema/metadata.

employee_id employee_name salary designation gender
101 monali 70000 developer 102 Amy 70000 developer 103 neha 65000 tester 104 amar 76000 developer male

I need to write this extra column along with the records where the extra columns are present without selecting the valid columns. This column is ever changing, so I can't define this column as "Gender" in the code.

Note: Here valid column means the columns which are determined by metadata/schema and invalid columns/extra columns can be defined as which are not present in metadata.

0 Answers
Related