Summarize PySpark data frame with first non-null value for each ID for all columns

Megha 0
Viewed 19

I have a PySpark data frame with the following structure but with many more rows and columns.

id col1 col2 col3
1 A null null
1 null B null
1 null null C
2 D null null
2 null E null
2 null null F

I want to summarize the data frame by selecting the first non-null value for each ID for all the columns.

The result I need:

id col1 col2 col3
1 A B C
2 D E F

I can do this in R with dplyr using the below code:

df %>% group_by(id) %>% summarize_all(list(~first(na.omit(.))))

Is there a way to do this in PySpark?

pyspark r
1 Answers

Try this:

import pyspark.sql.functions as f

df = (
    df
    .groupBy('id')
    .agg(*[f.first(f.col(column), ignorenulls= True).alias(column) for column in df.columns if column != 'id'])
)
ARCrow0
Related