Empty list representation in PySpark

user_12642588 0
Viewed 1083

I have a spark DataFrame with a column named "Ingredients". It has some values like:

['banana', 'apple']
['meat'] 
[]
[]

I want to look at only the []. Tried this:

display(df.filter(df.ingredients == []))

But got error:

java.lang.RuntimeException: Unsupported literal type class java.util.ArrayList []
pyspark apache-spark python
3 Answers

You can specify an empty array to compare:

import pyspark.sql.functions as F

display(df.filter(df.ingredients == F.array()))

Or you can check the array length is zero:

display(df.filter(F.size(df.ingredients) == 0))
user_141657300

Adding further to to @mck's answer, sometimes you have an array which contains only one empty string and it is also shown like 'empty array'. Here's an example :

df = spark.createDataFrame([([''],)], ['value'])

df.show()

# +-----+
# |value|
# +-----+
# |   []|
# +-----+

df.filter(F.col("value") == F.array(F.lit(""))).show()

# +-----+
# |value|
# +-----+
# |   []|
# +-----+

df.filter(F.col("value") != F.array(F.lit(""))).show()

# +-----+
# |value|
# +-----+
# +-----+

In this case F.col("value") == F.array() won't work.

user_13865510

try define like this code

import pyspark.sql.functions as F
import pyspark.sql.types as T
df = df.withColumn("ids",F.lit(None).astype(T.ArrayType(T.StringType())))

the ids will stored as None the ids's dtype is array<string>

and query with spark-sql like

select *  from tb1 where ids is not null
user_34803590
Related