Fastest way to check which rows in Numpy Matrix match value conditions?

Question

I have a 2D numpy array. Each row contains time-sampled data which must be matched to several different possible criteria.

For each row, several different criteria must be applied. Each row is data from a system, and each criteria represents a different situation that could be detected. A criteria condition is detected if select columns in the row are within a certain interval (greater than and less than certain values).

Is there a way to do this very fast and effectively in numpy? I have been looking around and didn't really find a way, although I am new to numpy.

I was thinking of a hash mechanism, such as storing some hash for each row and seeing if the criteria's hash matches that row's hash. However, the hash mechanism would only work with fixed values, and not ranges of values. Therefore, this would not be really applicable.

Example data:

import numpy as np
myData = np.array([[7.241, 7.123, 9.0, 1.2, 1.0, 1.5, 67.14, 162.32], [8.123, 10.4, 7.68, 1.6,0.7, 2.3, 21.2, 175.2]])

Example conditions:

Criteria #1: 
   condition 1 : column 1 must be greater than 6 and less than 7
   condition 2 : column 4 must be greater than 0 and less than 2
   condition 3 : column 6 must be greater than 10 and less than 50
   (conditions are 'and' operation, so all conditions must be met for criteria to be met or not)
Criteria #2: 
   condition 1 : column 3 must be greater than 10 and less than 22
   condition 2 : column 4 must be greater than 9 and less than 15

I was thinking of whether numpy supports some sort of regexing so to speak in this regard. Are there any mechanisms or techniques in numpy that would facilitate this efficiently?

The above example is simple, but in actual application, the matrix would contain several hundreds of rows and there could be several tens of criteria to evaluate against each row.

user_2988730 · Answer

First option: simple vectorized comparison:

c1 = (data[:, 0] > 6) & (data[:, 0] < 7) & (data[:, 3] > 0) & (data[:, 3] < 2) & (data[:, 5] > 10) & (data[:, 5] < 50)

c1 is a boolean mask of the locations where a row triggers criterion 1. You can use it as an index into the first dimension of data.

Second option: extract the columns of interest and compare against upper and lower criteria simultaneously:

c1lower = [6, 0, 10]
c1upper = [7, 2, 50]
c1cols = data[:, [0, 3, 5]] # this is a copy
c1 = ((c1cols > c1lower) & (c1cols < c1upper)).all(1)

Copying the data may be slower but combining it across the row should be faster.

In both cases, you can convert the boolean mask to a numerical index using np.where, np.nonzero, np.flatnonzero or np.argwhere.

Fastest way to check which rows in Numpy Matrix match value conditions?

1 Answers