Copy warning when filtering dataframe in pandas

user_7207358 0
Viewed 1480

I have a list of cars and their prices. I need to apply log transformation on the prices, but some prices are set to 0, so I first need to create a copy of my original df with the prices filtered out:

import pandas as pd
import numpy as np

cars = {'Brand': ['Honda Civic','Toyota Corolla','Ford Focus','Audi A4'],
        'Price': [22000,25000,27000,35000]
        }

df = pd.DataFrame(cars, columns = ['Brand', 'Price'])

print (df)

df2 = df.loc[(df.Price>=22000) & (df.Brand=='Honda Civic')]

print(df2)

df2['logprice']=np.log(df2['Price'])

I get the warning 

A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

But I'm already using .loc !

I've read everything about this warning but I still don't understand what I'm doing wrong...

pandas copy python
2 Answers

In order to make sure no warning add copy

df2 = df.loc[(df.Price>=22000) & (df.Brand=='Honda Civic')].copy()
df2['logprice']=np.log(df2['Price'])
user_79645270

But I'm already using .loc !

And it's your problem. Don't use it for assigning a copy of your filtering rows. Use

df2 = df[(df.Price>=22000) & (df.Brand=='Honda Civic')]

instead.

The explanation:

The problem arises not in the mentioned command, but in the subsequent one:

df2['logprice']=np.log(df2['Price'])

which is in its left side interpreted as

df.loc[(df.Price>=22000) & (df.Brand=='Honda Civic')]['logprice']

Note the dangerous ][ — it's something that should be avoided on the left side of the assignment operator, because it may do change (adding a column in your case) not into your dataframe, but only into some temporary object (which will be destroyed by garbage collector).

Using

  • .loc[] may provide a view or a copy (by context), while

  • [] (which calls .__getitem__() under the hood) always returns a copy, so the left side of the assignment

    df2['logprice']=np.log(df2['Price'])

will be not interpreted as I stated, but unambiguously as adding a new column to your real dataframe.

user_70235900
Related