ndarray contained in pandas dataframe is not copied when using df.copy()

user_15347765 0
Viewed 167

I have a Dataframe df1 with column 'a'. each row in column 'a' contains a 2d numpy array

> df1 
                                               a
0  [[10.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0,...
1  [[10.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0,...
2  [[10.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0,...
3  [[10.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0,...

I want to create a copy of this Dataframe and make changes to the 2d arrays without effecting the arrays in df1

So I took an explicit deep copy

df2 = df1.copy(deep=True)
> df2
                                               a
0  [[10.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0,...
1  [[10.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0,...
2  [[10.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0,...
3  [[10.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0,...

Now changing a value in the array in the first cell of df2 is also effecting df1

df2.loc[0,'a'][0,0] = 0
> df2
                                              a
0  [[0.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0, ...
1  [[0.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0, ...
2  [[0.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0, ...
3  [[0.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0, ...

> df1
                                              a
0  [[0.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0, ...
1  [[0.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0, ...
2  [[0.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0, ...
3  [[0.0, 1.0, 1.0], [1.0, 1.0, 1.0], [1.0, 1.0, ...

How can I avoid this and what is the best way to make a copy of all sub objects of a Dataframe? Thanks

pandas dataframe python-3.x numpy python
1 Answers

As pandas documentations says:

Note that when copying an object containing Python objects, a deep copy will copy the data, but will not do so recursively. Updating a nested data object will be reflected in the deep copy.

You have a 2d numpy array in each column.

To create a new deep copy recursively you can create another dataframe using copy.deepcopy for the data.

df2= pd.DataFrame(data=copy.deepcopy(df1.values), columns=df1.columns)
user_99647780
Related