Pyspark filter dataframe if column does not contain string

user_4208407 0
Viewed 12776

I hope it wasn't asked before, at least I couldn't find. I'm trying to exclude rows where Key column does not contain 'sd' value. Below is the working example for when it contains.

values = [("sd123","2"),("kd123","1")] 
columns = ['Key', 'V1']
df2 = spark.createDataFrame(values, columns)

df2.where(F.col('Key').contains('sd')).show()

how to do the opposite?

pyspark apache-spark-sql apache-spark python
1 Answers

Use ~ as bitwise NOT:

df2.where(~F.col('Key').contains('sd')).show()
user_141657300
Related