PySpark DataFrame Filter Column Contains Multiple Value

user_2009051 0
Viewed 6873

Just wondering if there are any efficient ways to filter columns contains a list of value, e.g:

Suppose I want to filter a column contains beef, Beef:

I can do:

beefDF=df.filter(df.ingredients.contains('Beef')|df.ingredients.contains('beef'))

Instead of doing the above way, I would like to create a list:

beef_product=['Beef','beef']

and do:

beefDF=df.filter(df.ingredients.contains(beef_product))

I don't need to maintain code but just need to add new beef (e.g ox, ribeyes) in the beef_product list to have the filter dataframe.

Obviously the contains function do not take list type, what is a good way to realize this?

pyspark apache-spark dataframe
2 Answers

Try with .isin() accepts list.

beefDF=df.filter(df.ingredients.isin(beef_product))

Example:

df=spark.createDataFrame([(1,'beef'),(2,'Beef'),(3,'b')],['id','ingredients'])

from pyspark.sql.functions import *
beef_product=['Beef','beef']
df.filter(df.ingredients.isin(beef_product)).show()
#+---+-----------+
#| id|ingredients|
#+---+-----------+
#|  1|       beef|
#|  2|       Beef|
#+---+-----------+
user_76326950 
from pyspark.sql.functions import *
df=spark.createDataFrame([(1,'beef'),(2,'Beef'),(3,'Cow'), (3,'Tiger')],  
                         ['id','ingredients'])
df.filter("ingredients in ('Beef','Tiger')").show()
user_140630920
Related