Find start/stop location of sharp cumulative events

Mitch Anthony 0
Viewed 30

Here is an example set of data:

EDIT: Included some more data.

x = [0, 5, 6,15, 20, 40, 73,
     100,101,102,103,104,105,106,108,111,115,
     116,117,118,119,120,123,124,125,126,127,
     128,129,130,131, 150,161,170, 183, 194,
     210, 234, 257, 271,272,273,274, 275,276,
     277,278,279,280,281,282,283,284,285,287,
     288,291,292,293,294,295,296,297,298,300,301,
     302,303,304,305,306,307,308,309,310,311,
     340, 351, 358, 360, 380, 390, 400,401,
     402,403, 404, 405, 408, 409, 413, 420,
     425,426,427,428,429,430,431,432,433,434,435,
     436, 440, 450, 455]

y = np.arange(1, len(x)+1)

enter image description here

Here is what the data visually looks like and has the potentially for each sharp increase to be longer. The last sharp increase also has a pause, but I would like it to still be considered one set of data. Black dots are the gradient.

I am attempting to find the the start/end x-values for each sharp increase in cumulative counts. So the output should be an array of indexes, like what Riley has done.

A vectorized method would be ideal to help with any time constraints to quickly go through data. Here is rough outline of what has been done so far within a pandas dataframe.

  • Shift the "x-data" and take a difference
  • See if sequential differences are below a threshold to create logic array
  • Do rolling sum on logic array with so Trues will continue add to count
  • Find when rolling sum exceeds another threshold
  • Compare with previous value to ensure it is increase/decreasing for start/stop times
  • Add times to index list

It seems a little finicky on some of the rolling averages and isn't as quick as I would like. Multiplying some of these large arrays with logic arrays seems to take a good amount of time.

EDIT: Here is the code Riley has provided and offers an excellent start. It is also only a couple lines a code, versus my method above was almost 50 or so.

rate_threshold = 0.25 
min_consecutive = 8 
above_rate = np.gradient(y,x) >= rate_threshold
sequence_diff = np.diff(np.lib.stride_tricks.sliding_window_view(above_rate, window_shape=min_consecutive).all(axis=1).astype(int))
intervals = np.hstack([np.argwhere(sequence_diff==1) + 1, np.argwhere(sequence_diff==-1) + min_consecutive-1])

The new issue comes from the final sharp increase of data. Two sets of start/end points are returned, where the desired would just be one.

My initial thought is to include some kind of average routine with the sliding window to account for these drops in the gradient so the end is so hard set.

pandas numpy python-3.x
1 Answers

Not sure what your desired output would look like, so let's start with this, verify it does it what you want it to, then go from there

rate_threshold = 1
min_consecutive = 5

above_rate = np.diff(y)/np.diff(x) >= rate_threshold
sequence_diff = np.diff(sliding_window_view(above_rate, window_shape=min_consecutive).all(axis=1).astype(int))
intervals = np.hstack([np.argwhere(sequence_diff==1) + 1, np.argwhere(sequence_diff==-1) + min_consecutive-1])

intervals is a 2d numpy array of indices whose 2 columns are first index, and last index in a sequence (of length min_consecutive) of rates above the threshold

array([[ 7, 12],
       [16, 20],
       [22, 29],
       [39, 52],
       [56, 62],
       [64, 74]], dtype=int64)
user9C8M0
Related