Overloading Pandas operators

user2655730 0
Viewed 42

I am trying to subclass a pandas Series and overload the operators, but cannot figure out why I'm getting a recursion error. Minimally reproduceable example below:

import pandas as pd

class MySeries(pd.Series):

    def __mul__(self, other):
        print('hello!')
        return super().mul(other)

MySeries([1,2,3]) * 1

I get the following error:

hello!
...
hello!
  File "/home/jibi/.local/lib/python3.9/site-packages/pandas/core/ops/__init__.py", line 197, in flex_wrapper
    return op(self, other)
  File "<stdin>", line 4, in __mul__
...
  File "/home/jibi/.local/lib/python3.9/site-packages/pandas/core/ops/__init__.py", line 197, in flex_wrapper
    return op(self, other)
  File "<stdin>", line 4, in __mul__
packages/pandas/core/ops/common.py", line 90, in get_op_result_name
    if isinstance(right, (ABCSeries, ABCIndex)):
RecursionError: maximum recursion depth exceeded

Observations:

  • the following don't seem to make a difference:
  • The error only occurs when I multiply the subclassed instance by an integer, but all is well if I multiply it by a Series or another subclassed instance
  • calling MySeries([1,2,3]).mul(1) also breaks. It doesn't break if I remove the __mul__ method in MySeries.
  • This happens with other operators (__add__, __sub__, etc)
  • I can subclass a DataFrame and override the __mul__ method with the exact same code and the * operator behaves as expected (i.e. this is only happening when I subclass a Series)

Been banging my head on the desk for a few hours now--any insight is appreciated! Thanks!

pandas version 1.4.3; python 3.9.5

pandas python
1 Answers

It is important to note that mul and __mul__ are two different functions in the pd.Series. The double underscore is mainly a convention in Python to represent private variables.

When you call MySeries.__mul__: first this results in a call of super().mul(..) whose internal logic is defined in pandas to call the __mul__ function of MySeries, which in turn calls super().mul(..) and so on and so on... hence the infinite recursion.

The solution is to replace super().mul(other) with super().__mul__(other)

The following code gives the desired result:

import pandas as pd

class MySeries(pd.Series):

    def __mul__(self, other):
        print('hello!')
        return super().__mul__(other)

MySeries([1,2,3]) * 2
Theo Charalambous0
Related