Filtering unique valued rows of a numpy array as much as possible

datatech 0
Viewed 35

I have a table like this;

table = np.array([[ 67, 15],
                  [ 90, 15],
                  [ 92, 15],
                  [ 67, 25],
                  [138, 25],
                  [138, 35],
                  [ 62, 15],
                  [ 70, 25],
                  [ 71, 25],
                  [124, 35]])

I want to select predetermined number of rows (target) which include unique values (never shown before) if it is possible . If it is not, selecting with same order by following the same logic.

I.e., if I want to select;

2 rows: [67,15], [138,25]
3 rows: [67,15], [138,25], [124,35] 
4 rows: [67,15], [138,25], [124,35] , [90,15]
5 rows: [67,15], [138,25], [124,35] , [90,15] , [67,25]

and so on.

Here is my trial;

space = []
id_space = [0]
space.append(table[0,:])
target  = 3
row = 1

for i in range(1,len(table)):
    if not (any(np.isin(table[i,:],np.hstack(space)))):
        space.append(table[i,:])
        id_space.extend([i])
        row =  row + 1

    if (row==target):
        break

table[id_space]

It works up until target = 3, but does not for the rest. By the way, matrix named table is really huge in reality. Maybe some faster alternatives are also available.

Thanks in advance!

filter numpy python
1 Answers

The following code should give you the expected answer you are looking for, if I have understood your requirements correctly. It was a quick rough attempt and can definitely be made more efficient as there are a lot of potentially repeated calculations and parts of the code that could benefit from some refactoring, perhaps at the expense of simplicity. See how it does with your table size and if needed benchmark to find slower parts of the code.

target  = 5
# This is a list of the row numbers in the order that they satisfy the logic 
row_indexes = []
# This array keeps track of values that we use to filter the table by
unique_values = np.array([])

while len(row_indexes) != target:
    
    # If there are no unique values to filter the table by we iterate 
    # through the table and choose the first row number that has not already 
    # been added to row_indexes, and the values in that row are used to filter
    # the rest of the table
    if unique_values.size == 0:

        for i in range(len(table)):

            if i not in row_indexes:
                row_indexes.append(i)
                unique_values = np.append(unique_values, table[i])
                break

    else:
        
        # Finds indexes of the rows in table where the entire row is unique using 
        # the unique_values array as the filter 
        bool_arr = np.all(np.isin(table, unique_values, invert=True), axis=1)
        unique_row_indexes = np.where(bool_arr)[0] # np.where returns a tuple of len 1 

        # If there are no more unique rows we reset the unique values and loop through again
        if unique_row_indexes.size == 0:
            unique_values = np.array([])
            continue

        else:
            
            exists_unique_row = False

            for row_index in unique_row_indexes: 
                
                # If the unique row has already been added skip it
                if row_index in row_indexes:
                    continue
                 
                row_indexes.append(row_index)
                unique_values = np.append(unique_values, table[row_index])
                exists_unique_row = True
                break
            
            # If all unique rows have already been added in a previous pass through
            # we reset the unique values and start again
            if not exists_unique_row:
                unique_values = np.array([])


print(table[row_indexes])
H_Boofer0
Related