filter rows for column value in list of words pyspark

user_11255139 0
Viewed 206

I am working with pyspark dataframe. I have a column words (array<string>)as shown below:

+---+--------------------------------------------------------------------------------+
| id|                                                                           words|                      
----+--------------------------------------------------------------------------------+
|012|[content, type, multipart, alternative, boundaries, nextpart, nextpart, drama,..|
|013|[received, from, am5eur02ht120, eop, eur02, prod, protection, outlook by, pro...|
|014|[data, care, much, important, information, summer, care, send, faraway, forget..|

Also I have a list of words:

list = ["protection", "content", "received"]

I want to filter rows which have any value from the list.

Expected output:

+---+--------------------------------------------------------------------------------+
| id|                                                                           words|                      
----+--------------------------------------------------------------------------------+
|012|[content, type, multipart, alternative, boundaries, nextpart, nextpart, drama,..|
|013|[received, from, am5eur02ht120, eop, eur02, prod, protection, outlook by, pro...|
pyspark apache-spark python
1 Answers

I think that you can use array_intersect function together with size function to achieve what you need, something like this (not tested, not sure about exact code for this F.lit(list)):

>>> import pyspark.sql.functions as F

>>> df.show()
+----------------+
|           words|
+----------------+
|[content, word2]|
|      [111, 222]|
+----------------+

>>> list_col = F.array(*[F.lit(cl) for cl in list])
>>> df.filter(F.size(F.array_intersect(F.col("words"), list_col)) > 0).show()
+----------------+
|           words|
+----------------+
|[content, word2]|
+----------------+
Alex_Ott_186270
Related