I have a dataframe which has a column containing an array of structs. I need to filter the array based on the value of one of the elements in those nested structs.
The first approach i used was a filter higher order function and passing through a lambda anonymous function.
df = df.withColumn("filtered", F.filter("nested_array", lambda x: x.some_column == 1))
However, I have found that using the F.expr function seems to be faster.
df = df.withColumn("filtered", F.expr("filter(nested array, x => x.some_column == 1")))
Is this because when using the lambda function, data has to be serialized / deserialized? Or something else?
Additionally, is there a way i can tell in the Spark UI if serialization / deserialization is happening?
Many thanks