Update a pyspark Delta Table using a python boolean function

Viewed 179

so I have a delta table that I want to update based on a condition of two column values combined;

i.e.

delta_table.update(
    condition=is_eligible(col("name"), col("age"))
    set={"pension_eligible": lit("yes")}
)

I'm aware that I can do something similar to:

delta_table.update(
    condition=(col("name") == "Einar") & (col("age") > 65)
    set={"pension_eligible": lit("yes")}
)

But since my logic for computing this is quite complex (I need to look up the name in a database) I would like to define my own Python function for computing this (is_eligible(...)). Other reasons are because this function is used elsewhere and I would like to minimize code duplication.

Is this possible at all? As I understand you could define it as an UDF, but they only take one parameter and I need at least two. I can not find anything about more complex conditions in the delta lake documentation, so I'd really appreciate some guidance here.

0 Answers
Related