Count the number of days from consecutive timeperiods (Python, datetime)

user_8510149 0
Viewed 567

I want to calculate the number of days from consecutive periods.

In the df below I have four columns:

  • id; representing a person.
  • period; a number where the lowest is the first period and the highest the latest.
  • in_date; date when the period starts.
  • out_date; date when the period ends.

I want to build a generic function that performs the following:

  • calculate the number of days of consecutive periods. Two periods are considered consecutive if the number of days between them is less than 90.

  • I want only to calculate the number of days if the last period for an id has an out_date in 2013. If the last period has an 'out_date' of 2014 or 2012 I want to disregard that ID.

  • I want to include the days between periods in the result variable.

My problem is, as I'm fairly new to Python, I can't seam to come up with a good idea how to calculate the days between periods and classify a consecutive period. Any help would be much appreciated.

import pandas as pd
import numpy as np
import datetime

data = {'id':[1, 1, 1, 2, 2, 2, 2, 3, 3, 3],
        'period':[1, 2, 3, 1, 3, 5, 6, 2, 3, 4],
       'in_date': ['2011-02-15','2011-11-10','2012-10-13',
                   '2010-04-03','2012-02-17','2012-08-15','2014-01-04','2010-06-01','2012-03-29','2012-09-12'],
       'out_date': ['2011-05-21','2012-10-11','2013-10-25',
                    '2012-02-16','2012-02-19','2013-11-23','2014-12-18','2011-08-21','2012-09-11','2013-01-10']}
df = pd.DataFrame(data)

df['in_date'] = pd.to_datetime(df['in_date'])
df['out_date'] = pd.to_datetime(df['out_date'])
df['n_days'] = df['out_date'] - df['in_date']

Expected output:

enter image description here

pandas datetime python
1 Answers

First, convert n_days into a numeric value and ensure that the df is sorted:

df['n_days'] = (df['out_date'] - df['in_date']).dt.days
df = df.sort_values(['id','period'])

Add a column counting the days between periods:

df['days_since_last'] = (df['in_date'] - df['out_date'].shift(1)).dt.days

...and make sure those values don't cross between different id values:

id_changed = (df['id'].shift(1) != df['id'])
df.loc[id_changed, 'days_since_last'] = np.nan

Define a condition that notes where the days between are too high:

days_cut = (df['days_since_last'] >= 90)

Grab a subset of the dataframe where it is either a new id or a valid consecutive run of days. Assign each of these valid runs a unique value of 'run' (to be used for grouping later):

tmp = df[days_cut | id_changed ].copy()
tmp['run'] = range(len(tmp))

Merge that back into the main dataframe and fill run forward so that it shows where the valid runs of sequential periods are:

df = pd.merge(df, tmp[['id','period','run']], on=['id','period'], how='left')
df['run'] = df['run'].fillna(method='ffill')

This is what it looks like at that point. You can see that for each id there are contiguous runs of run values:

print(df)
   id  period    in_date   out_date  n_days  days_since_last  run
0   1       1 2011-02-15 2011-05-21      95              NaN  0.0
1   1       2 2011-11-10 2012-10-11     336            173.0  1.0
2   1       3 2012-10-13 2013-10-25     377              2.0  1.0
3   2       1 2010-04-03 2012-02-16     684              NaN  2.0
4   2       3 2012-02-17 2012-02-19       2              1.0  2.0
5   2       5 2012-08-15 2013-11-23     465            178.0  3.0
6   2       6 2014-01-04 2014-12-18     348             42.0  3.0
7   3       2 2010-06-01 2011-08-21     446              NaN  4.0
8   3       3 2012-03-29 2012-09-11     166            221.0  5.0
9   3       4 2012-09-12 2013-01-10     120              1.0  5.0

Extract the consecutive days for each run by summing up the n_days column. The .agg also tracks the max date in the run, so we can keep only the runs that end in 2013:

consecutive_days = df.groupby(['id','run']).agg( {'n_days' : np.sum, 'out_date' : np.max } )
consecutive_days = consecutive_days[(consecutive_days['out_date'].dt.year == 2013)]

consecutive_days = consecutive_days.drop(columns=['out_date']).rename(columns={'n_days' : 'consecutive_days'})

Finally, merge that back in to the original dataframe and drop the excess columns:

df = pd.merge(df, consecutive_days, on='id', how='left')
df = df.drop(columns=['days_since_last','run'])

print(df)
   id  period    in_date   out_date  n_days  consecutive_days
0   1       1 2011-02-15 2011-05-21      95             713.0
1   1       2 2011-11-10 2012-10-11     336             713.0
2   1       3 2012-10-13 2013-10-25     377             713.0
3   2       1 2010-04-03 2012-02-16     684               NaN
4   2       3 2012-02-17 2012-02-19       2               NaN
5   2       5 2012-08-15 2013-11-23     465               NaN
6   2       6 2014-01-04 2014-12-18     348               NaN
7   3       2 2010-06-01 2011-08-21     446             286.0
8   3       3 2012-03-29 2012-09-11     166             286.0
9   3       4 2012-09-12 2013-01-10     120             286.0
user_49871310
Related