Pythonic way to transform date vector into ranges?

user_477883 0
Viewed 49

I have a pandas DataFrame with one row per day, and some boolean columns. I'd like to transform them into a DataFrame that holds the ranges where these columns are True.

An example starting DF:

import pandas as pd

t = True
f = False

df = pd.DataFrame(
    {'indic': [f, f, t, t, t, f, f, f, t, f, f, t, t, t, t]},
    index=pd.date_range("2018-01-01", "2018-01-15")
)

print(df)

            indic
2018-01-01  False
2018-01-02  False
2018-01-03   True
2018-01-04   True
2018-01-05   True
2018-01-06  False
2018-01-07  False
2018-01-08  False
2018-01-09   True
2018-01-10  False
2018-01-11  False
2018-01-12   True
2018-01-13   True
2018-01-14   True
2018-01-15   True

This DataFrame's column is True from 2018-01-03 to 2018-01-05, then on 2018-01-09 (only one day), and then again from 2018-01-12 to 2018-01-15.

The output I am looking for in this example is this DF (date objects instead of strings would be okay too, even preferred):

desired_result = pd.DataFrame({
    'from': ["2018-01-03", "2018-01-09", "2018-01-12"],
    'to': ["2018-01-05", "2018-01-09", "2018-01-15"]
})

print(desired_result)

         from          to
0  2018-01-03  2018-01-05
1  2018-01-09  2018-01-09
2  2018-01-12  2018-01-15

As an extension, in a follow-up step I'd like this to work for multiple columns, e.g.:

df = pd.DataFrame(
    {
        'indic_A': [f, f, t, t, t, f, f, f, t, f, f, t, t, t, t],
        'indic_B': [f, f, f, f, f, f, f, f, t, t, t, t, t, f, f]
    },
    index=pd.date_range("2018-01-01", "2018-01-15")
)

desired_result = pd.DataFrame({
    'from': ["2018-01-03", "2018-01-09", "2018-01-12", "2018-01-09"],
    'to': ["2018-01-05", "2018-01-09", "2018-01-15", "2018-01-13"],
    'what': ["indic_A", "indic_A", "indic_A", "indic_B"]
})

print(desired_result)

         from          to     what
0  2018-01-03  2018-01-05  indic_A
1  2018-01-09  2018-01-09  indic_A
2  2018-01-12  2018-01-15  indic_A
3  2018-01-09  2018-01-13  indic_B

Is there a pythonic, elegant way to do this - maybe even a pandas function?

pandas date datetime python
2 Answers

Use melt for reshape first, then create helper column for unique groups by cumsum, filter only Trues by boolean indexing and aggregate agg by functions first and last:

df = df.rename_axis('date').reset_index().melt('date', var_name='ind', value_name='boolean')
df['new'] = (~df['boolean']).cumsum()
df = (df[df['boolean']]
         .groupby('new')
         .agg({'date':['first','last'], 'ind':'first'})
         .reset_index(drop=True))
df.columns = df.columns.map('_'.join)
print (df)
  date_first  date_last ind_first
0 2018-01-03 2018-01-05   indic_A
1 2018-01-09 2018-01-09   indic_A
2 2018-01-12 2018-01-15   indic_A
3 2018-01-09 2018-01-13   indic_B
user_29010020

You could try pd.DataFrame.shift

First make 2 new columns of shift up and down

df['down_shift'] = df['indic'].shift()
df['up_shift'] = df['indic'].shift(-1)

And the df will be like

            indic down_shift up_shift
2018-01-01  False        NaN    False
2018-01-02  False      False     True
2018-01-03   True      False     True
2018-01-04   True       True     True
2018-01-05   True       True    False
2018-01-06  False       True    False
2018-01-07  False      False    False
2018-01-08  False      False     True
2018-01-09   True      False    False
2018-01-10  False       True    False
2018-01-11  False      False     True
2018-01-12   True      False     True
2018-01-13   True       True     True
2018-01-14   True       True     True
2018-01-15   True       True      NaN

The idea here is

  • case 1: (indic, down_shift) = (True, False) - start
  • case 2: (indic, up_shift) = (True, False) - end
  • case 3: both case 1 and case 2 happen - start and end

So we use the trick

  • True - False = 1
  • False - True = -1
  • True - True = 0
  • False - False = 0

Code:

case_start = df['indic'] - df['down_shift']
case_end = df['indic'] - df['up_shift']

start_date_list = df[case_start == 1].index
end_date_list = df[case_end == 1].index

Then we check start_date_list

DatetimeIndex(['2018-01-03', '2018-01-09', '2018-01-12'], dtype='datetime64[ns]', freq=None)

Then we check end_date_list

DatetimeIndex(['2018-01-05', '2018-01-09'], dtype='datetime64[ns]', freq='4D')

The last date doesn't change from True to False, so we need to add it manually.

user_25832530
Related