Here's a test pandas dataframe to reproduce the error in pandas >= 1.1.0
df = pd.DataFrame({'A':[1,2,5], 'B':['abc', 'cde', 4], 'C': [.01,4.5, 6.7]})
What works -
def custom_func(row):
if not isinstance(row[1], str):
return row[0] + row[2]
else:
return 0
Apply Function as below -
df.apply(lambda row: custom_func(row), axis=1, raw=True)
df.apply(lambda row: custom_func(row), axis=1)
The moment we return a tuple or multiple values from a custom apply function, it starts giving Value Error.
Doesn't Work -
def custom_func_does_not_work(row):
if not isinstance(row[1], str):
return row[0] + row[2], row[0]*row[2]
else:
return 0, row[0]
Apply Function as -
df.apply(lambda row: custom_func_does_not_work(row), axis=1, raw=True)
The moment I remove raw=True, It works fine.
df.apply(lambda row: custom_func_does_not_work(row), axis=1)
If you downgrade pandas to say 0.23.4, This works fine with raw=True.
The error that you see in the above case is -
ValueError: Shape of passed values is (3, 2), indices imply (3, 3)
I understand the error, but not sure if there is a hidden switch in pandas.DataFrame.apply that can help us return the frame with raw=True
The reason I'm using raw=True is for pure performance reason, above is a small example, the custom_func is much more complex in reality and hence would like to use raw=True.