pyspark calculate mean of all columns in one line

user_1714692 0
Viewed 9725

I would like to calculate the mean value of each column without specifying all the columns name.

So for example instead of doing:

res = df.select([mean('col1'), mean('col2')])

I would like to do something equivalent to:

res = df.select([mean('*')])

Is that possible?

pyspark mean python-3.x
2 Answers

You can do it by

res  = df.select(*[f.mean(c).alias(c) for c in df.columns])
user_46595300

similar solution but maybe a little easier to read:

results= df.agg(*(avg(c).alias(c) for c in df.columns))

to easily retrieve the information use:

results.first().asDict()

it will be handy when you use it to fill NaN like:

df.na.fill(results.first().asDict())

no promotion just gratitude from my side: I picked that cool trick up in a wonderfull pyspark-class by Layla AI (Pyspark Essentials for Data Scientists)

user_147785900
Related