DataFrame as default function argument not working as expected

user_13052866 0
Viewed 148

I'm confused why a DataFrame as a default function argument doesn't work quite as I expected. It puzzles me why I still have to specify it in the function call.

Basically I've created a function to merge dataframes to a main dataframe (which I set as the default argument), based on a key (like a LEFT JOIN)

Simplified Dataframes (Just the same student with different subject scores):

dfA = pd.DataFrame ({'student': ['A'],
                     'math_score': [50]})

dfB = pd.DataFrame({'student': ['A'],
                    'eng_score': [70]})

dfC = pd.DataFrame({'student': ['A'],
                    'sci_score': [80]})

Function (I want to get his subject scores one by one, by merging them to a main df)

def merge_selected(df_to_merge, main_df = dfA):

    # LEFT JOIN to main_df
    main_df = main_df.merge(right=df_to_merge, how='left', on=['student'])

    return main_df

Using Function to Merging Twice:

dfA = merge_selected(dfB)
dfA = merge_selected(dfC)

Now here is what I don't get. I lost eng_score on the 2nd merge. Somehow when I assigned dfA to merge_selected a second time, it was removed.

  student  math_score  sci_score
0       A          50         80

But if I specified main_df = dfA in the function call:

dfA = merge_selected(dfB, main_df = dfA)
dfA = merge_selected(dfC, main_df = dfA)

I don't lose eng_score and get all his scores:

  student  math_score  eng_score  sci_score
0       A          50         70         80

Essentially I've solved the issue, but was hoping someone could shed some light on this.

Why do I still have to specify main_df even though it is supposed to default to dfA?

Also, why do I lose eng_score if I do not specify main_df = dfA?

pandas dataframe python-3.x function python
1 Answers

When you define your function with default argument main_df=dfA, the DataFrame dfA is ‘remembered’ by the function for all future calls. Let’s give this ‘original form’ of dfA, as at the creation of the function, a name: orig_dfA.

Now, take your first call to merge_selected. You end up creating a new, merged DataFrame, using orig_dfA, which is stored somewhere in memory, and is distinct from orig_dfA.

You then return from the function and assign dfA to this new, merged DataFrame. Here’s the thing. You’ve not actually changed orig_dfA at all. You’ve simply made the name ‘dfA’ refer to this new, merged DataFrame, which is stored at a different location in memory to orig_dfA. But the function is still attached to orig_dfA, which hasn’t changed - the function has no idea that you want the default value of main_df to now be the new DataFrame that ‘dfA’ refers to!

So, when it comes to the second function call, it’s as if the first one never happened!

Here’s a simpler example to illustrate this point. Define your function with the default value of main_df, as before. Now, immediately after the definition of your function, change dfA to something that’s totally not a DataFrame, say dfA = 3. Now call your function as you did the first time. You’ll get the exact same output.

What’s going on here is a mix of two things:

  1. In Python, the assignment (=) operator simply binds a name to an object/value in memory. The statement x = 3 doesn’t work as ‘set aside some memory which will be referred to by the name “x”, and put the value 3 there’, which one might intuitively expect. Instead, it’s more like ‘create the value 3 somewhere in memory and make the name “x” refer to it’.
  2. When you set a default parameter, the object/value that that default parameter refers to is determined at function creation time and is FIXED.

In your case, setting main_df=df_A when you defined the function translated to ‘make main_df refer to the object/value in memory that df_A currently refers to’. From this point on, you can make df_A refer to any other piece of memory you like - the function will always refer to the same piece of memory that df_A referred to when the function was created, not whatever value df_A happens to refer to when the function is actually called.

user_119966650
Related