How to write polars custom apply function that does the processing row by row?

user_18601363 0
Viewed 768

I need to create a new column in my dataframe that stores the processed values. So I used polars apply function to do some processing of dicoms and then return value. But this apply function by default takes the entire column as polars Series and it doesn't process row by row.

df = df.with_columns(
        [
            pl.apply(
                exprs=["Filename", "Dicom_Tag", "Dicom_Tag_Corrected", "Name"],
                f=apply_corrections_polars,
            ).alias("dicom_tag_value_corrected"),
        ]
    )
python-polars apply python
1 Answers

As the documentation of pl.apply states, it should not be used in the select context. It should only be used in groupby operations to apply over different groups.

For custom functions over items, you can utilize the Struct data type.

Since polars>=0.13.16 you can apply over Struct dtypes. A Struct can be composed of any column in polars.

df = pl.DataFrame({"ham": [2, 2, 3], 
              "spam": [11, 22, 33], 
              "foo": [3, 2, 1]})

def my_complicated_function(struct: dict) -> int:
    """
    A function that can not utilize polars expressions.
    This should be avoided.
    """

    # do work
    return struct["ham"] + struct["spam"] + struct["foo"]

df.select([
    pl.struct(["ham", "spam", "foo"]).apply(my_complicated_function)
])


shape: (3, 1)
┌─────┐
│ ham │
│ --- │
│ i64 │
╞═════╡
│ 16  │
├╌╌╌╌╌┤
│ 26  │
├╌╌╌╌╌┤
│ 37  │
└─────┘
user_67170540
Related