pyspark.sql.utils.ParseException error when filtering the df

user_5928092 0
Viewed 115

I want to select all rows from a pyspark df except some rows where the array contains 1. It works with the code below in the notebook:

<pyspark df>.filter(~exists("<col name>", lambda x: x=="hello"))

But when I write it as this:

cond = '~exists("<col name>", lambda x: x=="hello")'
df = df.filter(con)

I got error as below:

pyspark.sql.utils.ParseException: 
extraneous input 'x' expecting {')', ','}(line 1, pos 32)

I really can't spot any typo. Could someone give me a hint if I missed something?

Thanks, J

pyspark
1 Answers

To pass in the conditions through variable, it needs to be written in the form of expr str of spark sql. So it can be modified to:

cond = '!exists(col_name, x -> x == "hello")'
user_170214290
Related