functools reduce In-Place modifies original dataframe

user_1833326 0
Viewed 718

I currently facing the issue that "functools.reduce(operator.iadd,...)" alters the original input. E.g.

I have a simple dataframe df = pd.DataFrame([[['A', 'B']], [['C', 'D']]])

        0
0  [A, B]
1  [C, D]

Applying the iadd operator leads to following result:

functools.reduce(operator.iadd, df[0])
['A', 'B', 'C', 'D']

Now, the original df changed to 

              0
0  [A, B, C, D]
1        [C, D]

Also copying the df using df.copy(deep=True) beforehand does not help.

Has anyone an idea to overcome this issue? THX, Lazloo

pandas functools dataframe python-3.x python
2 Answers

Use operator.add instead of operator.iadd:

In [8]: functools.reduce(operator.add, df[0])
Out[8]: ['A', 'B', 'C', 'D']

In [9]: df
Out[9]: 
        0
0  [A, B]
1  [C, D]

After all, operator.iadd(a, b) is the same as a += b. So it modifies df[0]. In contrast, operator.add(a, b) returns a + b, so there is no modification of df[0].

Or, you could compute the same quantity using df[0].sum():

In [39]: df[0].sum()
Out[39]: ['A', 'B', 'C', 'D']

The docs for df.copy warns:

When deep=True, data is copied but actual Python objects will not be copied recursively, only the reference to the object.

Since df[0] contains Python lists, the lists are not copied even with df.copy(deep=True). This is why modifying the copy still affects df.

user_1905970

In addition to @unutbu's good answer, you can also use the int.__add__ method:

df = pd.DataFrame([[['A', 'B']], [['C', 'D']]])
functools.reduce(lambda x,y: (x).__add__(y), df[0])
print(df)

And You can see that it is:

        0
0  [A, B]
1  [C, D]

For output!!!

user_87083640
Related