spark dataframe groupping does not count nulls

Question

I have a spark DataFrame which is grouped by a column aggregated with a count:

df.groupBy('a').agg(count("a")).show

+---------+----------------+
|a        |count(a)        |
+---------+----------------+
|     null|               0|
|      -90|           45684|
+---------+----------------+


df.select('a').filter('aisNull').count

returns

warning: there was one feature warning; re-run with -feature for details
res9: Long = 26834

which clearly shows that the null values were not counted initially.

What is the reason for this behaviour? I would have expected (if nullat all is contained in the grouping result) to properly see the counts.

user_8534386 · Answer

value_counts(dropna=False) pyspark's equivalent:

from pyspark.sql import functions as f
df.groupBy('a').agg(f.count('*')).orderBy('count(1)',ascending=False).show()

spark dataframe groupping does not count nulls

3 Answers