pyspark continuous invocation of UDF when using between method filter

dataviews 0
Viewed 20

I'm experiencing a UDF being invoked continuously when I call the filter method on a DataFrame and use the between method as a condition.

UDF

@udf(returnType=FloatType())
def test_udf(value):
    calc = value * 2.00
    return calc

Adding a new column to my DF:

df.withColumn("value_of_cash", test_udf(lit(2)))

Now filtering with between:

df = df.filter(
    df.value_of_cash.between(0, df.spending_wise)
)

There isn't any error message, but I can see from logs that the UDF function gets invoked multiple times when using the between method call.

pyspark python-3.x python
1 Answers

Spark does lazy evaluation. That means it will listen to you and absolutely do nothing and continue to do until you ask for final answer.

In your example the between clause can only be executed when your column value_of_cash is computed. To compute value_of_cash it needs to run the UDF as many times equal to number of rows in your data. For spark, UDFs are black box and therefore it stupidly computes it for every row.

the_ordinary_guy0
Related