Find Start / Stop Index Range For Values in NumPy Array Greater Than N

Viewed 645

Let's say I have a NumPy array:

x = np.array([2, 3, 4, 0, 0, 1, 1, 4, 6, 5, 8, 9, 9, 4, 2, 0, 3])

For all values in x >= 2, I need to find the start / stop indices where consecutive values of x >=2 (i.e., a run of one single value greater than or equal to 2 is not counted). Then, I repeat this for x >= 3, x >=4, ..., x >= x.max(). The output should be a NumPy array three columns (first column is the minimum value, the second column is the inclusive start index, and the third column is the stop index) and will look like:

[[2,  0,  2],
 [2,  7, 14],
 [3,  1,  2],
 [3,  7, 13],
 [4,  7, 13],
 [5,  8, 12],
 [6, 10, 12],
 [8, 10, 12],
 [9, 11, 12]
]

Naively, I could look through each unique value and then search for the start/stop indices. However, this requires doing multiple passes over x. What's the best NumPy vectorized way to accomplish this task? Is there a solution that doesn't require multiple passes over the data?

Update

I realized that I also need to count the single instances as well. So, my output should be:

[[2,  0,  2],
 [2,  7, 14],
 [2, 16, 16],  # New line needed
 [3,  1,  2],
 [3,  7, 13],
 [3, 16, 16],  # New line needed
 [4,  2,  2],  # New line needed
 [4,  7, 13],
 [5,  8, 12],
 [6,  8,  8],  # New line needed
 [6, 10, 12],
 [8, 10, 12],
 [9, 11, 12]
]
2 Answers

Here's another solution (which I believe can be improved):

import numpy as np
from numpy.lib.stride_tricks import as_strided

x = np.array([2, 3, 4, 0, 0, 1, 1, 4, 6, 5, 8, 9, 9, 4, 2, 0, 3])

# array of unique values of x bigger than 1
a = np.unique(x[x>=2])

step = len(a)  # if you encounter memory problems, try a smaller step
result = []
for i in range(0, len(a), step):
    ai = a[i:i + step]
    c = np.argwhere(x >= ai[:, None])
    c[:,0] = ai[c[:,0]]
    c =  np.pad(c, ((1,1), (0,0)), 'symmetric')

    d = np.where(np.diff(c[:,1]) !=1)[0]

    e = as_strided(d, shape=(len(d)-1, 2), strides=d.strides*2).copy()
    # e = e[(np.diff(e, axis=1) > 1).flatten()]
    e[:,0] = e[:,0] + 1 

    result.append(np.hstack([c[:,0][e[:,0, None]], c[:,1][e]]))

result = np.concatenate(result)

# array([[ 2,  0,  2],
#        [ 2,  7, 14],
#        [ 2, 16, 16],
#        [ 3,  1,  2],
#        [ 3,  7, 13],
#        [ 3, 16, 16],
#        [ 4,  2,  2],
#        [ 4,  7, 13],
#        [ 5,  8, 12],
#        [ 6,  8,  8],
#        [ 6, 10, 12],
#        [ 8, 10, 12],
#        [ 9, 11, 12]])

Sorry for not commenting what each step does -- if later I will find time I will fix it.

This is quite interesting problem, indeed. I attempted to solve it splitting it into three parts.

Grouping:

import numpy as np
import pandas as pd
x = np.array([2, 3, 4, 0, 0, 1, 1, 4, 6, 5, 8, 9, 9, 4, 2, 0, 3])
groups = pd.DataFrame(x).groupby([0]).indices

So groups is dictionary {0: [3, 4, 15], 1: [5, 6], 2: [0, 14], 3: [1, 16], 4: [2, 7, 13], 5: [9], 6: [8], 8: [10], 9: [11, 12]} and its values are numpy arrays of dtype=int64.

Masking:

In this part I iterate through several masks arrays x>=i for each unique value i in descending order:

mask_array = np.zeros(x.size).astype(int)
for group in list(groups)[::-1]:
    mask = mask_array[groups[group]] = 1
    # print(group, ':', mask_array)
    # output = find_slices(mask)

And these masks look like this:

9 : [0 0 0 0 0 0 0 0 0 0 0 1 1 0 0 0 0]
8 : [0 0 0 0 0 0 0 0 0 0 1 1 1 0 0 0 0]
6 : [0 0 0 0 0 0 0 0 1 0 1 1 1 0 0 0 0]
5 : [0 0 0 0 0 0 0 0 1 1 1 1 1 0 0 0 0]
4 : [0 0 1 0 0 0 0 1 1 1 1 1 1 1 0 0 0]
3 : [0 1 1 0 0 0 0 1 1 1 1 1 1 1 0 0 1]
2 : [1 1 1 0 0 0 0 1 1 1 1 1 1 1 1 0 1]
1 : [1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 0 1]
0 : [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1]

Extracting slices from masks:

I expect to construct some function called find_slices that extracts slice positions from mask arrays (if you uncomment it). This is what I did:

def find_slices(m):
    m1 = np.r_[0, m]
    m2 = np.r_[m, 0]
    starts, = np.where(~m1 & m2)
    ends, = np.where(m1 & ~m2)
    return np.c_[starts, ends - 1]

For example, slice positions of array [0 1 1 0 0 0 0 1 1 1 1 1 1 1 0 0 1] would be [[1, 2], [7, 13], [16, 16]]. Note that this is not a standard way of returning slices, end position is usually incremented by 1.

Final script

After all, one needs some manoeuvres to make the expected output, here is like it looks at the end:

import numpy as np
import pandas as pd
x = np.array([2, 3, 4, 0, 0, 1, 1, 4, 6, 5, 8, 9, 9, 4, 2, 0, 3])
groups = pd.DataFrame(x).groupby([0]).indices
mask_array = np.zeros(x.size).astype(bool)

m = []
for group in list(groups)[::-1]:
    mask_array[groups[group]] = True
    s = find_slices(mask_array)
    group_output = np.c_[np.repeat(group, s.shape[0]), s] #insert first column
    m.append(group_output) 
output = np.concatenate(m[::-1])
output = output[output[:,1]!= output[:,2]] #elimate slices with unit length

Output:

 [[ 0  0 16]
 [ 1  0  2]
 [ 1  5 14]
 [ 2  0  2]
 [ 2  7 14]
 [ 3  1  2]
 [ 3  7 13]
 [ 4  7 13]
 [ 5  8 12]
 [ 6 10 12]
 [ 8 10 12]
 [ 9 11 12]]
Related