Remove a list from a list of lists in PySpark DataFrame

user_4584909 0
Viewed 225

I have the following dataframe, with a list column, that contains a list of lists. In some specific cases, this list contains one unique list, with contains only empty strings.

+--+---------------+
|id|list           |
+--+---------------+
|1 |[[1, a],[2, b]]|
|2 |[[,]]          |

And I would like to filter out this list, so that I have the following:

+--+---------------+--------------+
|id|list           |result        |
+--+---------------+--------------+
|1 |[[1, a],[2, b]]|[[1, a],[2, b]|
|2 |[[,]]          |[]            |

I have tried the following:

data_frame.withColumn("result", array_remove("list", ["",""]))

but I have the following error:

An error occurred while calling z:org.apache.spark.sql.functions.array_remove. Unsupported literal type class java.util.ArrayList [, ]

Any suggestions or ideas?

pyspark apache-spark-sql apache-spark python
1 Answers

It's a bit tricky here - I think using a raw Spark SQL query will be appropriate:

df2 = data_frame.withColumn('result', F.expr("array_remove(list, array('',''))"))

df2.show()
+---+----------------+----------------+
| id|            list|          result|
+---+----------------+----------------+
|  1|[[1, a], [2, b]]|[[1, a], [2, b]]|
|  2|          [[, ]]|              []|
+---+----------------+----------------+
user_141657300
Related