New DataFrame is being made when updating dataframe inside a loop

Question

I am trying to make some changes to three dataframes in a loop in this manner.

for sheet in [f1, f2, f3]: 
    sheet = preprocess_df(sheet)

The preprocess_df function looks like this

def preprocess_df(df): 
    """ Making a function to preprocess a dataframe individually rather then all three together """
    
    # make column names uniform
    columns = [
        "Reporting_Type",
        "AA_name",
        "Date_DD/MM/YYYY",
        "Time_HHMMSS",
        "Type",
        "Name",
        "FI_Type",
        "Count_linked",
        "Average_timelag_FI_Notification",
        "FI_Ready_to_FI_request_ratio",
        "Count_Consent_Raised",
        "Actioned_to_raised_ratio",
        "Approved_to_raised_ratio",
        "FI_Ready_to_FI_request_ratio(Daily)",
        "Daily_Consent_Requests_Data_Delivered",
        "Total_Consent_Requests_Data_Delivered",
        "Consent_Requests_Data_Delivered_To_Raised_Ratio",
        "Daily_Consent_Requests_Raised",
        "Daily Consent_Requests_Data_Delivered_To_Raised_Ratio",
    ]
    
    # Set the sheet size 
    df = df.iloc[:, :19]
    
    # Set the column names 
    df.columns = columns

    return df

I am basically updating the column names and fixing the dataframe size. The issue that I face is that the sheet variable does get updated if I print the dataframe inside the loop, however, the original f1, f2 and f3 dataframes don't get updated. I think this is because the sheet variable creates a copy of f1 etc. rather than actually using the same dataframe. This seems like an extension of pass-by-reference or pass-by-value concept. Is there a way I could make inplace changes to all the sheets inside the loop?

user_4086503 · Answer

Indeed, a copy of the dataframe is created when you are doing df = df.iloc[:, :19]. However, you can get around this by using drop, with inplace=True:

import pandas as pd
import numpy as np

def preprocess_df(df): 
    columns = [
        "a",
        "b",
    ] # Swap this list with yours
    df.drop(df.columns[:2],inplace=True, axis=1) # Replace 2 with 19 in your code
    df.columns = columns

f1 = pd.DataFrame(np.arange(12).reshape(3, 4),columns=['A', 'B', 'C', 'D']) # Just an example
preprocess_df(f1) # You can put this in your for loop
print(f1)

The above code will output something like:

New DataFrame is being made when updating dataframe inside a loop

1 Answers