Calculate total number of null values in a dataset in PySpark

Alphonse Prakash 21MCA0062 0
Viewed 15

I have to calculate the total number of null values in my dataset including all rows and all columns and the output must be an integer that would represent the total number of null values in my dataset.

bigdata pyspark apache-spark null apache-spark-sql
1 Answers
null_cnt = df.select(
    F.aggregate(
        F.array(*[F.sum(F.when(F.isnull(c), 1)) for c in df.columns]),
        F.expr("0L"),
        lambda sum, x: sum + x
    )
).head()[0]

Test:

from pyspark.sql import functions as F
df = spark.createDataFrame([(1, 2), (None, None), (3, 4)], ['col1', 'col2'])

null_cnt = df.select(
    F.aggregate(
        F.array(*[F.sum(F.when(F.isnull(c), 1)) for c in df.columns]),
        F.expr("0L"),
        lambda sum, x: sum + x
    )
).head()[0]

print(null_cnt)
# 2
ZygD0
Related