Consider the following dataframe (see end of this answer for code to generate it):
A T0 T1
0 0 2017-01-02 2017-01-04
1 2 2017-01-02 2017-01-05
2 1 2017-01-03 2017-01-04
3 3 2017-01-04 2017-01-07
4 2 2017-01-07 2017-01-10
5 0 2017-01-08 2017-01-10
6 3 2017-01-08 2017-01-09
7 1 2017-01-10 2017-01-11
8 0 2017-01-11 2017-01-13
9 3 2017-01-12 2017-01-15
10 2 2017-01-13 2017-01-16
11 1 2017-01-15 2017-01-17
12 0 2017-01-18 2017-01-20
13 3 2017-01-19 2017-01-20
14 1 2017-01-20 2017-01-22
15 2 2017-01-20 2017-01-21
16 2 2017-02-03 2017-02-06
17 1 2017-02-03 2017-02-06
18 0 2017-02-04 2017-02-07
19 3 2017-02-05 2017-02-07
20 1 2017-02-07 2017-02-08
21 3 2017-02-09 2017-02-11
22 0 2017-02-09 2017-02-10
23 1 2017-02-13 2017-02-16
24 3 2017-02-15 2017-02-17
25 2 2017-02-15 2017-02-18
26 0 2017-02-17 2017-02-18
27 2 2017-02-19 2017-02-21
28 3 2017-02-20 2017-02-21
29 2 2017-02-24 2017-02-27
30 1 2017-02-25 2017-02-26
31 0 2017-02-27 2017-03-01
I want to group using a pandas.Grouper (e.g. by month) and on A and for each group g compute the maximum value of g.T0 - g.T1.shift().
I am currently doing the following:
def fun(x):
x['G'] = x.T0 - x.T1.shift()
return x
u = df.groupby((pd.Grouper(freq='MS', key='T0'), 'A')).apply(fun)
u = df.groupby((pd.Grouper(freq='MS', key='T0'), 'A'))['G'].max()
This works but is very slow — My current dataframe contains ~80m lines and it takes more than 30 minutes to compute this for the group I want.
The expected output for the above data frame would be:
T0 A
2017-01-01 0 5 days
1 6 days
2 4 days
3 4 days
2017-02-01 0 9 days
1 9 days
2 9 days
3 4 days
Name: G, dtype: timedelta64[ns]
I have tested another approach which consist in computing the gap before end:
df = df.sort_values(['A', 'T0'])
df['G'] = df.T0 - df.T1.shift()
df.loc[df['A'].diff() != 0, 'G'] = pd.NaT
Then I could simply:
df.groupby((pd.Grouper(freq='MS', key='T0'), 'A'))['G'].max()
The problem here is that the first entry of each group is the gap between the current group and the previous one, so I get:
T0 A
2017-01-01 0 5 days
1 6 days
2 4 days
3 4 days
2017-02-01 0 15 days
1 12 days
2 13 days
3 16 days
Name: G, dtype: timedelta64[ns]
What I would need would be:
df.groupby((pd.Grouper(freq='MS', key='T0'), 'A')).all_but_first()['G'].max()
The problem is that all_but_first does not exists in pandas.
Is there a way to either:
- Speed up the first code using
apply; - Compute for each group the maximum values, without regarding the first value for each group?
Code to generate the dataframe:
import numpy as np
import pandas as pd
A = np.repeat(range(4), 8)
T0 = [1, 7, 10, 17, 34, 39, 47, 57,
2, 9, 14, 19, 33, 37, 43, 55,
1, 6, 12, 19, 33, 45, 49, 54,
3, 7, 11, 18, 35, 39, 45, 50]
T1 = [3, 9, 12, 19, 37, 40, 48, 59,
3, 10, 16, 21, 36, 38, 46, 56,
4, 9, 15, 20, 36, 48, 51, 57,
6, 8, 14, 19, 37, 41, 47, 51]
df = pd.DataFrame({'A': A, 'T0': T0, 'T1': T1})
df['T0'] = pd.to_datetime(df['T0'], unit='D', origin=pd.Timestamp('2017-01-01'))
df['T1'] = pd.to_datetime(df['T1'], unit='D', origin=pd.Timestamp('2017-01-01'))
df = df.sort_values('T0')
df = df.reset_index(drop=True)