How to check if a condition lasted for more than 15 mins?

Viewed 305

Below is an example of the dataset

Date Value
2020-01-01 01:35 50
2020-01-01 01:41 49
2020-01-01 01:46 50

I wish to check if the 'Value' was equal to 50 for continuous period of 15 mins. If yes, I want to extract the date for which it occurred. Let me give an example what I mean by continuous period of 15 mins. Assume that I want to check if the value is equal to 50 for a continuous period of 5 mins (instead of 15 mins). The data that would satisfy this condition would be as follows

Date Value
2020-01-01 01:35 50
2020-01-01 01:36 50
2020-01-01 01:37 50
2020-01-01 01:38 50
2020-01-01 01:39 50

Then I want to extract the date2020-01-01 onto a list because the above data was equal to 50 for a continuous period of 5 mins (or more).

2 Answers

I am posting code for 5 mins so that output matches your desired output. Change 300 to 900 for 15 mins. Steps:

  1. Convert the df['Date'] to datetime so that we can subtract two dates to know the time difference between them.

  2. Group the df by date and Call f for each group object.

  3. In f: max-continuous_range gives the length of longest segment where value is 50. f return True if length is 5 mins or more. Append date in list if f returns True.

Use:

def f(g):
   mask = (g['Value'] == 50)
   max_continuous_range = (np.max(np.cumsum(g['Date'].where(mask).diff()))
                         + timedelta(minutes = 1))
   return  max_continuous_range.seconds >= 300

df['Date'] = pd.to_datetime(df['Date'])
groups = df.groupby(df['Date'].dt.date, as_index = False)
final_list = [str(idx) for idx, g in groups if f(g)]

Input:

    Date  Value
0  2020-01-01 01:35     40
1  2020-01-01 01:36     50
2  2020-01-01 01:37     50
3  2020-01-01 01:38     50
4  2020-01-01 01:39     50
5  2020-01-01 01:40     50
6  2020-01-01 01:41     40
7  2020-01-01 01:42     40

Output:

>>> final_list
['2020-01-01']

Inside f(g):

mask: True where value is 50.

    0    False
    1     True
    2     True
    3     True
    4     True
    5     True
    6    False
    7    False

df['Date'].where(mask) Puts NaT where mask is not True.

0                   NaT
1   2020-01-01 01:36:00
2   2020-01-01 01:37:00
3   2020-01-01 01:38:00
4   2020-01-01 01:39:00
5   2020-01-01 01:40:00
6                   NaT
7                   NaT

.diff gives difference between two consecuting elements. It will give NaT if any value is NaT. Result after df['Date'].where(mask).diff():

0               NaT
1               NaT
2   0 days 00:01:00
3   0 days 00:01:00
4   0 days 00:01:00
5   0 days 00:01:00
6               NaT
7               NaT

Now cumulative sum of difference between consecutive times will give us the total time elapsed. After np.cumsum(...):

0               NaT
1               NaT
2   0 days 00:01:00
3   0 days 00:02:00
4   0 days 00:03:00
5   0 days 00:04:00
6               NaT
7               NaT

np.max gives us the longest length. 1 minute is added to take care of boundary condition

You can create a dummy variable that will essentially act as an id of continuous runs. The logic behind it is comparing each value with the next one in the series to see if the results are not equal. Then, we cast them to integers and make a cumulative sum. This means that the dummy variable will only increase when a streak ends. We can then groupby on this dummy variable, apply count, and filter based on your condition

df['dummy_id'] = (df['Value'] != df['Value'].shift()).cumsum()
grouped_df = df.groupby("dummy_id").agg({"dummy_id": "count", "date": "min"})

All you have to do then is filter according to what you were looking for and you should be good to go.

Related