For example,
I have a pyspark data frame like this:
value
["9.1","0","0","-1.5","-3","9.456"]
["9.456","4","5.345","6","7.34","8"]
["7","4","5","6","7"]
["-1.66"]
["0"]
["-5","4","6"]
["9","4.567"]
The output should be:
value
["9.456","4","5.345","6","7.34","8"]
["7","4","5","6","7"]
["0"]
["9","4.567"]
The output should remove the rows where any of the array elements has a value less than 0.
I tried this: but, it didn't work:
is_greater = lambda x: x>0 == 0
df_new = df_previous.withColumn("value", filter(f.col("value"), is_greater))