Filtering a column with an empty array in Pyspark

user_427069 0
Viewed 1284

I have a DataFrame which contains a lot of repeated values. An aggregated, distinct count of it looks like below

> df.groupby('fruits').count().sort(F.desc('count')).show()


| fruits        | count       |
| -----------   | ----------- |
| [Apples]      | 123         |
| []            | 344         |
| [Apples, plum]| 444         |

My goal is to filter all rows where the value is either [Apples] or [].

Suprisingly, the following works for an non-empty array but for empty it doesn't

import pyspark.sql.types as T

is_apples = F.udf(lambda arr: arr == ['Apples'], T.BooleanType())
df.filter(is_apples(df.fruits).count() # WORKS! shows 123 correctly.

is_empty = F.udf(lambda arr: arr == [], T.BooleanType())
df.filter(is_empty(df.fruits).count() # Doesn't work! Should show 344 but shows zero.

Any idea what I am doing wrong?

pyspark apache-spark-sql apache-spark python
2 Answers

It might be an array containing an empty string:

is_empty = F.udf(lambda arr: arr == [''], T.BooleanType())

Or it might be an array of null:

is_empty = F.udf(lambda arr: arr == [None], T.BooleanType())

To check them all at once you can use:

is_empty = F.udf(lambda arr: arr in [[], [''], [None]], T.BooleanType())

But actually you don't need a UDF for this, e.g. you can do:

df.filter("fruits = array() or fruits = array('') or fruits = array(null)")
user_141657300

You can do it by checking the length if the array.

is_empty = F.udf(lambda arr: len(arr) == 0, T.BooleanType())
df.filter(is_empty(df.fruits).count()
user_56602840
Related