Create subset of multiple pandas dataframes using for statement

user_15381822 0
Viewed 22

I am working with multiple dataframes in pandas, and am looking to only select certain columns from each of them.

AUD = [AUD2yr,AUD5yr,AUD10yr,AUD30yr]

for df in AUD:
    df = df[['Date','Open']]

I would ideally like to keep the names of the dataframes, but i receive the following error:

'NoneType' object is not subscriptable

Any help would be much appreciated.

pandas dataframe python
1 Answers

'NoneType' object is not subscriptable

This means in your program somewhere None[...] is attempted. From the snippet you shared, that happens with df[...], i.e., somehow df became None.

Apart from that, doing df = df[["Date", "Open"]] inside the loop body will not alter the frames in the list; rather, it makes df now "look" at a new frame; the reference to the frame in the list is lost.

To fix that, you can traverse the list along with the index and modify with indexing:

for idx, frame in enumerate(AUD):
    AUD[idx] = frame[["Date", "Open"]]

or perhaps better, with a list comprehension:

AUD = [frame[["Date", "Open"]] for frame in AUD]

Lastly, to avoid the infamous SettingWithCopyWarning in case you further modify the frames, it's better you put .copy() after subsetting, e.g.,

AUD = [frame[["Date", "Open"]].copy() for frame in AUD]

Additionally, to affect the individual variables inside the list, i.e., AUD*yr vars in AUD = [AUD2yr, AUD5yr, AUD10yr, AUD30yr], there are some ways:

  • We can go for inplace=True option in operations we do. Note that this is not always possible; but in your specific case, we'll use .drop method which supports this, and we will drop all but "Date" and "Open" columns, so here we go:
for idx, frame in AUD:
    cols_to_drop = frame.columns.difference(["Date", "Open"])
    AUD[idx].drop(columns=cols_to_drop, inplace=True)

^this approach is infeasible with a list comprehension. But that brings us to 2nd option:

  • If there is handful of frames in the list, like you have 4 in AUD, we can unpack the new list again to those variables:
AUD2yr, AUD5yr, AUD10yr, AUD30yr = AUD = [frame[["Date", "Open"]].copy()
                                          for frame in AUD]

This will now not only alter AUD but also the individual variables. First option also does that, so your choice :)

  • There's a 3rd alternative... But that requires you to change AUD to be a dictionary rather than a list. This way you won't refer to the frames as bareword variables like AUD10yr but rather as dictionary keys, e.g., AUD["AUD10yr"]. For what it's worth, this might be a better practice if you got a lot frames especially. So,
# initialize as dictionary
AUD = {"AUD2yr": AUD2yr, "AUD5yr": AUD5yr, "AUD10yr": AUD10yr, "AUD30yr": AUD30yr}

# modify via dict comprehension, akin to list comprehension in the very above
AUD = {frame_name: frame[["Date", "Open"]].copy()
       for frame_name, frame in AUD.items()}

# now, can refer to individuals as:
print(AUD["AUD5yr"])

Some references:

user_93321870
Related