Obtain a view of a DataFrame using the loc method

user_11779593 0
Viewed 216

I am trying to obtain a view of a pandas dataframe using the loc method but it is not working as expected when I am modifying the original DataFrame.
I want to extract a row/slice of a DataFrame using the loc method so that when a modification is done to the DataFrame, the slice reflects the change.

Let's have a look at this example:

import pandas as pd
import numpy as np
df = pd.DataFrame({'ID':np.arange(0,5,2), 'a':np.arange(3), 'b':np.arange(3)}).set_index('ID')
df
    a   b
ID      
0   0   0
2   1   1
4   2   2

Now I create a slice using loc:

slice1 = df.loc[[2],]
slice1

    a   b
ID      
2   1   1

Then I modify the original DataFrame:

df.loc[2, 'b'] = 9
df

    a   b
ID      
0   0   0
2   1   9
4   2   2

But unfortunately our slice does not reflect this modification as I would be expecting for a view:

slice1
    a   b
ID      
2   1   1

My expectation:

    a   b
ID      
2   1   9

I found an ugly fix using a mix of iloc and loc but I hope there is a nicer way to obtain the result I am expecting.
Thank you for your help.

pandas dataframe python
3 Answers

Disclaimer: This is not an answer.

I tried testing how over-writing the values in chained assignment vs .loc referring to the pandas documentation link that was shared by @Quang Hoang above.

This is what I tried:

dfmi = pd.DataFrame([list('abcd'),
   list('efgh'),
   list('ijkl'),
   list('mnop')],
   columns=pd.MultiIndex.from_product([['one', 'two'],
   ['first', 'second']]))

df1 = dfmi['one']['second']
df2 = dfmi.loc[:, ('one', 'second')]

Output of both df1 and df2:

0    b
1    f
2    j
3    n

Iteration 1:

value = ['z', 'x', 'c', 'v']
dfmi['one']['second'] = value

Output df1:

0    z
1    x
2    c
3    v

Iteration 2:

value = ['z', 'x', 'c', 'v']
dfmi.loc[:, ('one', 'second')] = value

Output df2:

0    z
1    x
2    c
3    v

The assignment of new sets is changing the values in both the cases.

The documentation says:

Quote 1: 'method 2 (.loc) is much preferred over method 1 (chained [])'

Quote 2: 'Outside of simple cases, it’s very hard to predict whether "getitem" (used by chained option) will return a view or a copy (it depends on the memory layout of the array, about which pandas makes no guarantees), and therefore whether the "setitem" (used by .loc) will modify dfmi or a temporary object that gets thrown out immediately afterward.'

I am not able to understand the explanation above. If the value in dfmi can change (in my case) and may not change (like in Benoit's case) then which way to obtain the result? Not sure if I am missing a point here. Looking for help

user_126961630

The reason the slice didn't reflect the changes you made in the original dataframe is b/c you created the slice first.

When you create a slice, you create a "copy" of a slice of the data. You're not directly linking the two.

The short answer here is that you have two options 1) changed the original df first, then create a slice 2) don't slice, just do your operations referencing the original df using .loc or iloc

user_138137960

The memory address of your dataframe and slice are different, so changes in dataframe won't reflect in the slice-

Different memory addresses

The answer is to change the value in the dataframe and then slice it -

Correct output

user_100298090
Related