Numpy and The best way to remove rows with idendical values

user_6081863 0
Viewed 85

I'm struggling with numpy lib.

I have a tensor of the shape (batch_size, timestep, feature):

For example lets create a dummy:

x = np.arange(42).reshape(2,7,3)
#now make some rows have homogeneous values
x[:,::3,:] =0
x[:,::5,:] =2

Now I need a numpyish way(which is repeatable in tensorflow) to remove rows(axis=-2) where all values are the same. So in the end I need a tensor to look like this:

[[[ 3  4  5]
  [ 6  7  8]
  [12 13 14]]

 [[24 25 26]
  [27 28 29]
  [33 34 35]]]

Thanks. P.S. this is not the same question as to "remove all zero rows". Since here we are talking about rows with homo- values. And this is a bit trickier.

tensorflow numpy
2 Answers

If you are okay with losing one dimension (so that your array remains homogeneous), then you can do:

x[~np.all(x == x[:, :, 0, np.newaxis], axis=-1)]
# out:
[[ 3  4  5]
 [ 6  7  8]
 [12 13 14]
 [24 25 26]
 [27 28 29]
 [33 34 35]]

Credit: @unutbu's answer to a similar problem, here adapted to one more dimension.

Why is the 3rd dimension removed? Imagine if your conditions were such that you wanted to select 2 rows from your first array and 3 from your second: then the result would be heterogeneous, which would have to be stored as a masked array or as a list of arrays.

user_7581740

There might be a more clever way using only numpy. However, you could just iterate over the 2nd dimension and do a comparison.

not_same= []
for n in range(x.shape[1]): # iterate over the 2nd dimension
    # test if it is homogeneous i.e. first value equal all values
    not_same.append(~np.all(x[:,n,:] ==x[0,n,0]))
out = x[:,not_same,:]

This gives you:

array([[[ 3,  4,  5],
        [ 6,  7,  8],
        [12, 13, 14]],

       [[24, 25, 26],
        [27, 28, 29],
        [33, 34, 35]]])
user_11066340
Related