raw=True causes ValueError in pandas DataFrame.apply

Viewed 170

Here's a test pandas dataframe to reproduce the error in pandas >= 1.1.0

df = pd.DataFrame({'A':[1,2,5], 'B':['abc', 'cde', 4], 'C': [.01,4.5, 6.7]})

What works -

def custom_func(row):
    if not isinstance(row[1], str):
        return row[0] + row[2]
    else:
        return 0

Apply Function as below -

df.apply(lambda row: custom_func(row), axis=1, raw=True)
df.apply(lambda row: custom_func(row), axis=1)

The moment we return a tuple or multiple values from a custom apply function, it starts giving Value Error.

Doesn't Work -

def custom_func_does_not_work(row):
    if not isinstance(row[1], str):
        return row[0] + row[2], row[0]*row[2]
    else:
        return 0, row[0]

Apply Function as -

df.apply(lambda row: custom_func_does_not_work(row), axis=1, raw=True)

The moment I remove raw=True, It works fine.

df.apply(lambda row: custom_func_does_not_work(row), axis=1)

If you downgrade pandas to say 0.23.4, This works fine with raw=True.

The error that you see in the above case is -

ValueError: Shape of passed values is (3, 2), indices imply (3, 3)

I understand the error, but not sure if there is a hidden switch in pandas.DataFrame.apply that can help us return the frame with raw=True

The reason I'm using raw=True is for pure performance reason, above is a small example, the custom_func is much more complex in reality and hence would like to use raw=True.

0 Answers
Related