How to "describe" a column in pandas for python

Viewed 133

I'm using pandas in python and I have a dataframe where one column is a timestamp and others contain data.Example

The blue line stays constant for a while, then suddenly increases to zero, then at some point descends again to about -98 and stays there until it suddenly goes up to zero. What I need is a new column with the status of the blue color: constant,sudden increase, constant, decrease, constant, sudden increase, constant or somehow an object that describes the data:

blue line{ 
'08.02.2022 08:30:00.000' : 'sudden increase',
'08.02.2022 10:39:30.000' : 'start decrease',
'08.02.2022 10:59:40.000' : 'end decrease',
'08.02.2022 13:50:30.000' : 'sudden increase' 
}

Is there a package for something like this? I hope it isn't too far feched

Kind Regards, Alexander

2 Answers

constant,sudden increase, constant, decrease, constant, sudden increase, constant

Then I suggest taking looking at numpy.gradient, consider following simple example

import numpy as np
arr = np.array([0,0,0,1,2,3,4,5,6,7,7,7,7,7,5,3,1])
arrg = np.gradient(arr)
for a,g in zip(arr,arrg):
    print(a,g,sep=",")

output

0,0.0
0,0.0
0,0.5
1,1.0
2,1.0
3,1.0
4,1.0
5,1.0
6,1.0
7,0.5
7,0.0
7,0.0
7,0.0
7,-1.0
5,-2.0
3,-2.0
1,-2.0

Observe that constant parts result in zeros (potentially excluding these point adjacent to increase/decrease), increase result in positive values, decrease result in negative values, more increase/decrease the more value is far from zero.

You might need to convert pandas.Series (column of pandas.DataFrame) into numpy.array - for which pandas.Series sports method .to_numpy.

Disclaimer: this solution assumes that your data are evenly spaced, e.g. from sensor providing value each second.

Edit in order to detect borders of increase/decrease you might leverage numpy.diff following way, using arr and arrg from example above

arrb = abs(np.diff(arrg)) > 0.01
for a,b in zip(arr,arrb):
    print(a,b,sep=",")

output

0,False
0,True
0,True
1,False
2,False
3,False
4,False
5,False
6,True
7,True
7,False
7,False
7,True
7,True
5,False
3,False

Beware that due to how numpy.diff work you will get result shorter by 1 observation.

This does compute difference between adjacent elements then checks if it is absolute value is greater than 0.01 (you might need to adjust value depending on your data), when there is True (possibly few adjacent Trues) there is kink.

Here is how your data looks like.

In blue the data, in red the gradient. As the time interval is uniform, you can directly use the diff to compute the gradient (here clipped to a max of 1).

data

Now you can use numpy.select to map thresholds in the gradient to your labels.

Then, in pandas it is easy to filter the rows by changes in labels:

# input data
df = pd.read_excel('test.xlsx').set_index('time')
df.index = pd.to_datetime(df.index)

# compute gradient
grad = df['Col1'].diff()
# set labels
thresh_sudden = 10
thresh_gradual = 0.25
df['state'] = np.select([grad>thresh_sudden, grad<-thresh_gradual],
                        ['sudden increase', 'start decrease'],
                        'constant')  # default

# add end of decrease
df.loc[df['state'].eq('start decrease') & df['state'].shift(-1).eq('constant'),
       'state'] = 'end decrease'

# find changing points
change = df['state'].ne(df['state'].shift())
df['change'] = df['state'].where(change)

# slice
df[change]

output:

                      Col1      Col2            state           change
time                                                                  
2022-08-02 08:00:00 -98.05       NaN         constant         constant
2022-08-02 08:29:50 -96.99       NaN  sudden increase  sudden increase
2022-08-02 08:30:10  -0.01       NaN         constant         constant
2022-08-02 10:39:30  -0.89  1.351546   start decrease   start decrease
2022-08-02 10:59:20 -97.86  1.415426     end decrease     end decrease
2022-08-02 10:59:30 -98.04  1.415426         constant         constant
2022-08-02 13:50:20 -98.09  0.441630  sudden increase  sudden increase
2022-08-02 13:50:50  -0.12  0.441630         constant         constant
Related