For for Numpy array with zeros and ones, how to remove rows with duplicate ones?

user_10334421 0
Viewed 53

Usually in training neural networks with Keras for example, we need to create a X feature array and also a Y target array. And the Y target array should be one-hot encoded. For instance, X and Y can be something like this:

Y = np.array([[1,0,0],[1,0,0],[0,1,0],[0,1,0],[0,0,1],[0,0,1]])
X = np.array([[3,3,1],[3,3,2],[2,2,2],[2,2,1],[1,1,5],[1,2,4]])

Say if you like to consider only training examples with one unique target, thus we need to remove rows with duplicate ones in Y and remove the similar rows in X correspondingly, so you will end up with:

Y = np.array([[1,0,0],[0,1,0],[0,0,1]]) 
X = np.array([[3,3,1],[2,2,2],[1,1,5]])

How do we do so?

keras duplicates numpy python
2 Answers

Apply the following code:

for i in Y.T:
    index = 0
    is1 = 0
    for j in i:
        if j==1:
            is1+=1
            if is1>1:
                i[index]=2
        index+=1

Should give:

Y = np.array([[1,0,0],[2,0,0],[0,1,0],[0,2,0],[0,0,1],[0,0,2]])

Then removing rows with duplicate ones in Y and correspondingly with X is just:

Y = Y[(Y!=2).all(axis=1)]
X = X[(Y!=2).all(axis=1)]
user_103344210

Use np.unique(*, axis=0, retrun_index = True)

_, ix = np.unique(Y, axis = 0, return_index = True)

X[ix[::-1]]  # for your requested order
Out[]: 
array([[3, 3, 1],
       [2, 2, 2],
       [1, 1, 5]])

Y[ix[::-1]]
Out[]: 
array([[1, 0, 0],
       [0, 1, 0],
       [0, 0, 1]])
user_44277770
Related