Python: How to select columns that contain specific months of a year?

Viewed 470

I have a dataframe that looks like this:

+------------+-----------+-----+-----+----------+----------+-----+------------+----------+----------+-----+------------+
| State_Name | City_Name | ID1 | ID2 | 1/1/2020 | 1/2/2020 | ... | 12/31/2020 | 1/1/2021 | 1/2/2021 | ... | 12/31/2021 |
+------------+-----------+-----+-----+----------+----------+-----+------------+----------+----------+-----+------------+
| ...        | ...       | ... | ... | ...      | ...      | ... | ...        | ...      | ...      | ... | ...        |
+------------+-----------+-----+-----+----------+----------+-----+------------+----------+----------+-----+------------+
| ...        | ...       | ... | ... | ...      | ...      | ... | ...        | ...      | ...      | ... | ...        |
+------------+-----------+-----+-----+----------+----------+-----+------------+----------+----------+-----+------------+

I have a lot of columns from 1/1/2020 all the way to 12/31/2021. How do I select columns that contain specific months of a year? For example, if I want select columns that contain July 2021 data, I can subset columns with name '7/1/2021', '7/2/2021'......'7/31/2021'?

Thanks so much for your help!

3 Answers

You can use filter method:

df.filter(regex='7/\d{1,2}/2021', axis=1)

I suggest store all non datetime columns to MultiIndex, convert all another columns to datetimes:

print (df)
  State_Name City_Name ID1 ID2  7/1/2021  1/7/2020  7/05/2021  1/1/2021  \
0          a         b   s   d         7         8          5         6   

   1/2/2021  12/31/2021  
0         3           8  

df = df.set_index(['State_Name','City_Name','ID1','ID2'])
df.columns = pd.to_datetime(df.columns)

And for select july 2021 use DataFrame.loc with DatetimeIndex.strftime:

df1 = df.loc[:, df.columns.strftime('%m-%Y') == '07-2021']

Or compare by month periods by DatetimeIndex.to_period:

df1 = df.loc[:, df.columns.to_period('m') == '2021-07']
print (df1)
                              2021-07-01  2021-07-05
State_Name City_Name ID1 ID2                        
a          b         s   d             7           5

Or:

df1 = df.loc[:, (df.columns.year == 2021) & (df.columns.month == 7)]
print (df1)
                              2021-07-01  2021-07-05
State_Name City_Name ID1 ID2                        
a          b         s   d             7           5

One nice solution could be to use melt to transform the various date columns into values in a single date columns.

For example:

# make dataframe from sample data
data = {
    "State_Name": ['state1', 'state2'],
    "City_Name": ['city1', 'city2'],
    "ID1": ['ID1_A', 'ID1_B'],
    "ID2": ['ID2_A', 'ID2_B'],
    "1/1/2020": ['dog', 'cat'],
    "1/2/2020": ['house', 'mouse']
}
df = pd.DataFrame(data)

# melt date columns into a row
melted_df = df.melt(
    id_vars=["State_Name", "City_Name", "ID1", "ID2"],
    var_name="date")

df looks like this:

State_Name City_Name ID1 ID2 1/1/2020 1/2/2020
0 state1 city1 ID1_A ID2_A dog house
1 state2 city2 ID1_B ID2_B cat mouse

And melted_df looks like this:

State_Name City_Name ID1 ID2 date value
0 state1 city1 ID1_A ID2_A 1/1/2020 dog
1 state2 city2 ID1_B ID2_B 1/1/2020 cat
2 state1 city1 ID1_A ID2_A 1/2/2020 house
3 state2 city2 ID1_B ID2_B 1/2/2020 mouse

This makes it a bit more natural to select data based on dates. For instance, you could mark the date column as type datetime and select rows based on a date range.

start_date = pd.to_datetime('1/1/2020')
end_date = pd.to_datetime('1/1/2020')

# select rows where date is between given start/end date (inclusive)
sel_df = melted_df.loc[
    (melted_df['date'] >= start_date) & (melted_df['date'] <= end_date)
]

If you want to undo the melt, and the dates into columns again, you can use pivot. Eg.,

sel_df.pivot(columns='date', values='value')
Related