How to filter in rows where any column is null in pyspark dataframe

user_4745124 0
Viewed 1368

It has to be somewhere on stackoverflow already but I'm only finding ways to filter the rows of a pyspark dataframe where 1 specific column is null, not where any column is null.

import pandas as pd
import pyspark.sql.functions as f
my_dict = {"column1":list(range(100)),"column2":["a","b","c",None]*25,"column3":["a","b","c","d",None]*20}
my_pandas_df = pd.DataFrame(my_dict)

sparkDf = spark.createDataFrame(my_pandas_df)
sparkDf.show(5)

enter image description here

I'm trying to include any row with null values on any column of my dataframe, basically the opposite of this:

sparkDf.na.drop()
pyspark dataframe filter null
1 Answers

For including rows having any columns with null:

sparkDf.filter(F.greatest(*[F.col(i).isNull() for i in sparkDf.columns])).show(5)

For excluding the same:

sparkDf.na.drop(how='any').show(5)
user_98406370
Related