why does pd.drop not function the same outside a for-loop as it does inside a for loop?

user_6952783 0
Viewed 85

On a single dataframe, I can drop columns using the conventional df = df.drop('column name'). But when I try to loop over multiple dataframes and apply drop() to each one, the changes are not persistent. I know there is an inplace='True' argument that I can use but I am confused by what is fundamentally going on inside the for loop.

Example:

df_1 = pd.DataFrame({'A':[1,2,3], 'B':[4,5,6]})
df_1
    A   B
0   1   4
1   2   5
2   3   6

df_2 = pd.DataFrame({'A':[10,20,30], 'C':[40,50,60]})
df_2
     A   C
0   10  40
1   20  50
2   30  60

# this is the behavior I am looking for. 
df_1 = df_1.drop('A', axis=1)
df_1


    B
0   4
1   5
2   6

# when I put 2 dataframes in a for loop, I do not get the same output. 
df_1 = pd.DataFrame({'A':[1,2,3], 'B':[4,5,6]})
df_2 = pd.DataFrame({'A':[10,20,30], 'C':[40,50,60]})
full_data = [df_1, df_2]

# I expect this code to apply the "drop" for each of these dataframes in the same way 
# as above without the need for the "inplace" argument.
for dataset in full_data:
    dataset = dataset.drop('A', axis=1)

# the column 'A' should have been dropped for each dataframe while inside the loop 
# but it wasnt. why?
df_1
    A   B
0   1   4
1   2   5
2   3   6
pandas python
3 Answers

By doing dataset = dataset.drop('A', axis=1) in your loop, you are just assigning the variable in the loop. If you add print(dataset) you will see Column A dropped.

Try dataset.drop(columns=['A'], axis=1, inplace=True) in your loop instead.

for dataset in [df_1, df_2]:
    dataset.drop('A', axis=1, inplace=True)
user_109766540

The issue doesn't have to do with loop scoping specifically, but is a basic python assignment rules issue. See the following:

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'A': [1, 2], 'B': [3, 4]})

In [3]: another = df

In [4]: another is df
Out[4]: True

In [5]: another = another.drop('A', axis=1)

In [6]: another is df
Out[6]: False

In this example, you can see that assigning the result of the drop operation to another assignes a new object to the identifier another. It does not modify the df object in-place. On the other hand, using the inplace=True keyword does:

In [7]: another = df

In [8]: another.drop('A', axis=1, inplace=True)

In [9]: another is df
Out[9]: True

Essentially, there is no way to do what you are trying to do, which is to loop over a list of objects and then modify the object contents in place by re-assigning to the variables using the loop identifier. The reason the inplace=True argument works is because it is referencing a method on the dataframe itself, giving pandas control over the assignment of the result.

Check out this article on variables and object references or more info.

user_38887190

The object you are changing has to be defined outside of loop or else its scope is local to that for loop. This example uses a predefined list to solve the issue:

df_1 = pd.DataFrame({'A':[1,2,3], 'B':[4,5,6]})
df_2 = pd.DataFrame({'A':[10,20,30], 'C':[40,50,60]})
l=[]

for dataset in [df_1, df_2]:
    l.append(dataset.drop("A", axis=1))

df_1=l[0]
df_2=l[1]
user_163173000
Related