Pandas - Count consecutive rows with column values greater than a threshold limit

user_14168447 0
Viewed 1979

I have a dataframe where the speed of several persons is recorded on a specific time frame. Below is a simplified version:

df = pd.DataFrame([["Mary",0,2.3], ["Mary",1,1.8], ["Mary",2,3.2],
                   ["Mary",3,3.0], ["Mary",4,2.6], ["Mary",5,2.2],
                   ["Steve",0,1.6], ["Steve",1,1.7], ["Steve",2,2.5],
                   ["Steve",3,2.7], ["Steve",4,2.3], ["Steve",5,1.8],
                   ["Jane",0,1.9], ["Jane",1,2.7], ["Jane",2,2.3],
                   ["Jane",3,1.9], ["Jane",4,2.2], ["Jane",5,2.1]],
                   columns = [ "name","time","speed (m/s)" ])

print(df)
     name  time (s)  speed (m/s)
0    Mary         0          2.3
1    Mary         1          1.8
2    Mary         2          3.2
3    Mary         3          3.0
4    Mary         4          2.6
5    Mary         5          2.2
6   Steve         0          1.6
7   Steve         1          1.7
8   Steve         2          2.5
9   Steve         3          2.7
10  Steve         4          2.3
11  Steve         5          1.8
12   Jane         0          1.9
13   Jane         1          2.7
14   Jane         2          2.3
15   Jane         3          1.9
16   Jane         4          2.2
17   Jane         5          2.1

I'm looking for a way to count, for each name, how many times the speed is greater than 2 m/s for 2 consecutive records or more, and the average duration of these lapse times. The real dataframe has more than 1.5 million rows, making loops unefficient.

The result I expect looks like this:

     name  count  average_duration(s)
0    Mary      1                    4     # from 2 to 5s (included) - 1 time, 4/1 = 4s
1   Steve      1                    3     # from 2 to 4s (included) - 1 time, 3/1 = 3s
2    Jane      2                    2     # from 1 to 2s & from 4 to 5s (included) - 2 times, 4/2 = 2s

I've spent more than a day on this problem, without success... Thanks by advance for your help!

pandas count python
3 Answers

So here's my go:

df['over2'] = df['speed (m/s)']>2
df['streak_id'] = (df['over2'] != df['over2'].shift(1)).cumsum()
streak_groups = df.groupby(['name','over2','streak_id'])["time"].agg(['min','max']).reset_index()
positive_streaks = streak_groups[streak_groups['over2'] & (streak_groups['min'] != streak_groups['max'])].copy()
positive_streaks["duration"] = positive_streaks["max"] - positive_streaks["min"] + 1
result = positive_streaks.groupby('name')['duration'].agg(['size', 'mean']).reset_index()
print(result)

Output:

    name  size  mean
0   Jane     2     2
1   Mary     1     4
2  Steve     1     3

I'm basically giving each False/True streak a unique ID to be able to group by it, so each group is such a consecutive result.
Then I simply take the duration as the max time - min time, get rid of the streaks of len 1, and then get the size and mean of grouping by the name.

If you want to understand each step better, I suggest printing the intermediate DataFrames I have along the way.

user_50523650

Here is another version which checks for the condition (greater then 2) and creates a helper series s to keep track of duplicates later, then using series.where and series.duplicated we group on name using this result and aggregate count and nunique (number of unique values) , then divide:

c = df['speed (m/s)'].gt(2)
s = c.ne(c.shift()).cumsum()
u = (s.where(c&s.duplicated(keep=False)).groupby(df['name'],sort=False)
                                        .agg(['count','nunique']))

out = (u.join(u['count'].div(u['nunique']).rename("Avg_duration")).reset_index()
        .drop("count",1).rename(columns={"nunique":"Count"}))

print(out)

    name  Count  Avg_duration
0   Mary      1           4.0
1  Steve      1           3.0
2   Jane      2           2.0
user_98406370

Interesting question! I found it quite difficult to come up with a nice solution using pandas, but if you happen to know R and the dplyr package, then you could write something like this:

library(tidyverse)

df %>% 
  mutate(indicator = `speed_(m/s)` > 2.0) %>% 
  group_by(name) %>% 
  mutate(streak = cumsum(!indicator)) %>% 
  group_by(streak, .add = TRUE) %>% 
  summarise(duration = sum(indicator)) %>% 
  filter(duration >= 2) %>% 
  summarise(count = n(), mean_duration = mean(duration))

#> # A tibble: 3 x 3
#>   name  count mean_duration
#>   <chr> <int>         <dbl>
#> 1 Jane      2             2
#> 2 Mary      1             4
#> 3 Steve     1             3

Created on 2020-08-31 by the reprex package (v0.3.0)

I apologize in advance if this is too off-topic, but I thought that other R-users (or maybe pandas-wizards) would find it interesting.

user_77787870
Related