I have a dataframe and I would like to drop all rows with NULL value in one of the columns (string). I can easily get the count of that:
df.filter(df.col_X.isNull()).count()
I have tried dropping it using following command. It executes but the count still returns as positive
df.filter(df.col_X.isNull()).drop()
I tried different attempts but it returns 'object is not callable' error.
if you want to drop any row in which any value is null, use
df.na.drop() //same as df.na.drop("any") default is "any"
to drop only if all values are null for that row, use
df.na.drop("all")
to drop by passing a column list, use
df.na.drop("all", Seq("col1", "col2", "col3"))
another variation is:
from pyspark.sql.functions import col
df = df.where(col("columnName").isNotNull())
you can add empty string condition also somtimes
df = df.filter(df.col_X. isNotNull() | df.col_X != "")
You can use expr() functions that accept SQL-like query syntax.
from pyspark.sql.functions import expr
filteredDF = rawDF.filter(expr("col_X is not null")).filter("col_Y is not null")