Pyspark aggregation using dictionary with countDistinct functions

user_2571607 0
Viewed 695

I am trying to run aggregation on a dataframe. Then I want to calculate the distinct values on every column. I generate a dictionary for aggregation with something like:

from pyspark.sql.functions import countDistinct

expr = {x: "countDistinct" for x in df.columns if x is not 'id'}
df.groupBy("id").agg(expr).show()

I get error:

AnalysisException: Undefined function: 'countdistinct'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.;

If I use 'countDistinct' directly it works:

df.groupBy("id").agg(countDistinct('hours'))

Out[1]: DataFrame[id: int, count(hours): bigint]

This does not work:

df.groupBy("id").agg({'hours':'countDistinct'}).show()

AnalysisException: Undefined function: 'countdistinct'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.;

Any thoughts on how to work around this?

pyspark dataframe group-by sql
1 Answers

Seems that countDistinct is not a 'built-in aggregation function'.

Passing the distinct counted columns directly to agg would solve this:

cols = [countDistinct(x) for x in df.columns if x != 'id']

df.groupBy('id').agg(*cols).show()
user_47003270
Related