Model decay in pandas data frame

user_13929659 0
Viewed 61

I have the following pandas data frame:

user day value value_cumulative
Allen 2021-01-01 1 1
Allen 2021-02-01 2 3
Allen 2021-03-01 3 6
Allen 2021-04-01 4 10
Karen 2021-01-01 5 5
Karen 2021-02-01 6 11
Karen 2021-03-01 7 18
Karen 2021-04-01 8 26

Based on the above, I would like to add two columns that calculate decay in two different ways.

Version 1

This one is fairly straight forward. I simply want to take the current value plus the past values multiplied by the decay factor of 0.1.

user day value value_cumulative decay_v1 formula
Allen 2021-01-01 1 1 1.000 = 1
Allen 2021-02-01 2 3 2.100 = 2 + 0.1 * 1
Allen 2021-03-01 3 6 3.210 = 3 + 0.1 * 2.1
Allen 2021-04-01 4 10 4.321 = 4 + 0.1 * 3.21
Karen 2021-01-01 5 5 5.000 = 5
Karen 2021-02-01 6 11 6.500 = 6 + 0.1 * 5
Karen 2021-03-01 7 18 7.650 = 7 + 0.1 * 6.5
Karen 2021-04-01 8 26 8.765 = 8 + 0.1 * 7.65

Version 2

This one is a bit more complicated. I'm using the following formula:

decay_v2 = 1 - exp(-x*value_cumulative)

where x is defined as follows:

x = -log(1-sat)/center

where I have chosen sat = 0.05 and center as the value_cumulative from 2021-03-01. In the formula column below, the x is therefore:

 Allen: -log(1-0.05)/6  = 0.0085489

 Karen: -log(1-0.05)/18 = 0.0028496
user day value value_cumulative decay_v2 formula
Allen 2021-01-01 1 1 0.009 = 1-exp(-x*1)
Allen 2021-02-01 2 3 0.025 = 1-exp(-x*3)
Allen 2021-03-01 3 6 0.050 = 1-exp(-x*6)
Allen 2021-04-01 4 10 0.082 = 1-exp(-x*10)
Karen 2021-01-01 5 5 0.014 = 1-exp(-x*5)
Karen 2021-02-01 6 11 0.031 = 1-exp(-x*11)
Karen 2021-03-01 7 18 0.050 = 1-exp(-x*18)
Karen 2021-04-01 8 26 0.071 = 1-exp(-x*26 )

I have the feeling that it must be possible to somehow implement this using the ewm function (https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.ewm.html) but am unable to get it to work. Any ideas on how this could work?

pandas dataframe python
1 Answers

Managed to solve this with a function that implements a simple loop of the following type for decay_v1:

  decay_v1 = value.copy()
  
  # Replace all values except the first one, which is our starting point.
  for i in range(1, len(value)):
    decay_v1[i] = value[i] + 0.1 * decay_v1[i-1]

I applied this for all user names separately and then appended the results back together. For decay_v2 I solved it in a similar way.

user_139296590
Related