I have a large data frame with several columns which have similar names and are nested structs. I want to sum all of them up and get a final column with the same structure.
root
|-- employee1: struct (nullable = true)
| |-- first_week_salary: Decimal (nullable = true)
| |-- second_week_salary: Decimal (nullable = true)
| |-- third_week_salary: struct (nullable = true)
|-- day1_salary: Decimal (nullable = true)
| |-- day2_salary: Decimal (nullable = true)
|-- employee2: struct (nullable = true)
| |-- first_week_salary: Decimal (nullable = true)
| |-- second_week_salary: Decimal (nullable = true)
| |-- third_week_salary: struct (nullable = true)
|-- day1_salary: Decimal (nullable = true)
| |-- day2_salary: Decimal (nullable = true)
There are several more employee columns that start with the word 'employee' and have the same structure. I want to add all the employee columns to get a total column that has the same structure as the employee columns and the values are the addition of all employee values the above. Currently I have been able to add for 1 level by doing
field_list = [first_week_salary, second_week_salary, third_week_salary]
df = df.withColumn('total', f.struct(*[(df.employee1[field]+df.employee2[field]).alias(field) for field in field_list]))
However, this approach can only sum up columns up to 1 level, and also since we have many more employee columns, adding them this way seems very bulky. Is there a way to do this? Any help is appreciated.