Calculate cumulative sum forward pandas

Question

Suppose we have the following dataframe:

      Date    Type Country Value
0   2016-04-30  A   NL       1
1   2016-04-30  A   BE       2
2   2016-04-30  B   NL       3
3   2016-04-30  B   BE       4
4   2016-04-30  C   NL       5
5   2016-04-30  C   BE       6
6   2016-04-30  C   FR       7
7   2016-04-30  C   UK       8
8   2016-05-31  A   NL       9
9   2016-05-31  A   BE       10
10  2016-05-31  A   FR       11
11  2016-05-31  B   NL       12
12  2016-05-31  B   BE       13
13  2016-05-31  B   FR       14
14  2016-05-31  C   NL       15
15  2016-05-31  C   BE       16
16  2016-05-31  C   UK       17
17  2016-05-31  C   SL       18
18  2016-06-30  A   NL       19
19  2016-06-30  B   FR       20
20  2016-06-30  B   UK       21
21  2016-06-30  B   SL       22
22  2016-06-30  C   NL       23
23  2016-06-30  C   BE       24
24  2016-07-31  A   NL       25
25  2016-07-31  A   BE       23
26  2016-07-31  B   FR       12
27  2016-07-31  B   UK       28
28  2016-07-31  B   SL       22
29  2016-07-31  C   NL       25
30  2016-07-31  C   BE       28

Which corresponds to the following code:

df = pd.DataFrame([['2016-04-30','A','NL',1], ['2016-04-30','A', "BE" ,2], ['2016-04-30', 'B',  'NL',3], ['2016-04-30','B','BE',4], ['2016-04-30','C','NL',5], ['2016-04-30','C','BE',6],['2016-04-30','C','FR', 7], ['2016-04-30','C','UK',8], ['2016-05-31','A','NL',9], ['2016-05-31','A','BE',10], ['2016-05-31','A','FR',11], ['2016-05-31','B','NL',12], ['2016-05-31','B','BE',13], ['2016-05-31','B','FR',14], ['2016-05-31','C','NL',15], ['2016-05-31','C','BE',16], ['2016-05-31','C','UK',17], ['2016-05-31','C','SL',18], ['2016-06-30','A','NL',19], ['2016-06-30','B','FR',20], ['2016-06-30','B','UK',21], ['2016-06-30','B','SL',22], ['2016-06-30','C','NL',23], ['2016-06-30','C','BE',24], ['2016-07-31', 'A',   'NL', 25], ['2016-07-31', 'A', 'BE', 23], ['2016-07-31', 'B', 'FR',12], ['2016-07-31','B','UK',       28], ['2016-07-31','B', 'SL',22], ['2016-07-31',  'C',   'NL', 25], ['2016-07-31', 'C',   'BE',       28] ], columns=['Date','Type' ,'Country' ,'Value'])

I want to create an additional column 'CumValue', which computes the cumulative sum of the next K months (in this case lets say K=3, but I would like it to be general). So for example, for observation [2016-04-30, A, NL], I would want the CumValue to be 1 + 9 + 19 = 28 (so we include the initial month). Suppose for instance that the observation two months ahead is not available, then we set the value equal to NaN.

I would want the end product to look as follows:

      Date    Type Country Value  CumValue
0   2016-04-30  A   NL       1       29
1   2016-04-30  A   BE       2       NaN
2   2016-04-30  B   NL       3       NaN
3   2016-04-30  B   BE       4       NaN
4   2016-04-30  C   NL       5       43
5   2016-04-30  C   BE       6       46
6   2016-04-30  C   FR       7       NaN
7   2016-04-30  C   UK       8       NaN
8   2016-05-31  A   NL       9       53
9   2016-05-31  A   BE       10      NaN
10  2016-05-31  A   FR       11      NaN
11  2016-05-31  B   NL       12      NaN
12  2016-05-31  B   BE       13      NaN
13  2016-05-31  B   FR       14      46
14  2016-05-31  C   NL       15      63
15  2016-05-31  C   BE       16      68
16  2016-05-31  C   UK       17      NaN
17  2016-05-31  C   SL       18      NaN
18  2016-06-30  A   NL       19      NaN
19  2016-06-30  B   FR       20      NaN
20  2016-06-30  B   UK       21      NaN
21  2016-06-30  B   SL       22      NaN
22  2016-06-30  C   NL       23      NaN
23  2016-06-30  C   BE       24      NaN
24  2016-07-31  A   NL       25      NaN
25  2016-07-31  A   BE       23      NaN
26  2016-07-31  B   FR       12      NaN
27  2016-07-31  B   UK       28      NaN
28  2016-07-31  B   SL       22      NaN
29  2016-07-31  C   NL       25      NaN
30  2016-07-31  C   BE       28      NaN

Does anyone know an efficient way to do something like this?

user_5907936 · Answer

You can try the below code. I checked the output for (NL,A), (NL,C), (NL,BE), and it seems to work.

def shift_cum(x,k=3):
    return x.rolling(k).sum().shift(-2)

df.assign(CumValue=df.groupby(['Country','Type'])['Value'].apply(shift_cum))

Here we are passing the value , k in the function with default 3, which you can change when applying. The function first takes the rolling sum within the group and then shifts it back 2 positions to get match your requirement.

user_3944322 · Answer

Yati Raj's solution only works if all month values are continuously available. The OP states:

Suppose for instance that the observation two months ahead is not available, then we set the value equal to NaN

This is the case for Type 'A', Country 'BE': there are not data for 2016-06-30 available and hence the result should be NaN. In order to make it work for this case too, you can modify the solution as follows:

pd.merge(df, df.set_index('Date').groupby(['Type', 'Country']).Value.apply(lambda x: x.asfreq('1M').rolling(3).sum().shift(-2)).reset_index(), on=['Type', 'Country', 'Date']).rename(columns={'Value_x': 'Value', 'Value_y': 'CumValue'})

This yields the correct result for the second row as given in the OP:

         Date Type Country  Value  CumValue
0  2016-04-30    A      NL      1      29.0
1  2016-04-30    A      BE      2       NaN
...

(the accepted answer gave a CumValue of 35 here)

Calculate cumulative sum forward pandas

2 Answers