One hot encoding problem with p_utils.to_categorical

According to Keras documentation:

to_categorical: Converts a class vector (integers) to the binary class matrix.

Arguments: y: class vector to be converted into a matrix (integers from 0 to num_classes).

The input argument to the to_categorical take a list of integer that starts from zero. In your example, a list is [1, 2, 4, 5] which converted to 6 different classes (0 to 5):

[[0. 1. 0. 0. 0. 0.]
 [0. 0. 1. 0. 0. 0.]
 [0. 0. 0. 0. 1. 0.]
 [0. 0. 0. 0. 0. 1.]]

If you need the same number of classes you need a transformation which available in the sklearn package named LabelEncoder. It encodes labels with the value between 0 and n_classes-1. So if you pass your list, [1, 2, 4, 5] to LabelEncoder it converted to:

from sklearn import preprocessing
le = preprocessing.LabelEncoder()
print(le.fit_transform([1, 2, 4, 5]))
>>> [0 1 2 3]

As you can see, the labels start from zero which is what we needed for to_categorical method. The final step is to pass this list into to_categorical method.

new_labels = le.fit_transform([1, 2, 4, 5])
one_hot = to_categorical(new_labels)

>>> [[1. 0. 0. 0.]
    [0. 1. 0. 0.]
    [0. 0. 1. 0.]
    [0. 0. 0. 1.]]

Note: OneHotEncoder also exist in sklearn packages, but it needs LabelEncoder as well in such a case.

Thanks for your reply.

I have tried this (before your answer) which seems to work (around 0.997 on each epoch excepted for the first : 0.85) :

encoder = LabelEncoder()
encoder.fit(Y)
encoded_Y = encoder.transform(Y)
Y = np_utils.to_categorical(encoded_Y)

But accuracy seems to be 0 on test corpus :

model = load_model('model.h5')
# predict and evaluate
y_pred = model.predict_classes(X_test)
acc = np.sum(y_pred == Y_test) / np.size(y_pred)
print("Test accuracy = {}".format(acc))

Could you help me to find why ?

Model has good val_acc, I don't understand why accuracy test is so poor

EDIT : predict_classes returns classes like this : set([0, 1, 2, 3, 4, 5, 6, 7, 8]) but I compute accuracy on this : set([1, 2, 4, 5, 9, 12, 14, 19, 20])

How tell to keras to encode correctly ?

EDIT : problem solved with this :

encoder = LabelEncoder()
encoder.fit(Y_test)
Y_test = encoder.fit_transform(Y_test) here

I get 99.0574712644 % accuracy

Train on 10584 samples, validate on 2646 samples Epoch 1/30 10584/10584 [==============================] - 315s 30ms/step - loss: 1.4155 - acc: 0.4357 - val_loss: 0.5511 - val_acc: 0.7827 Epoch 2/30 5152/10584 [=============>................] - ETA: 2:16 - loss: 0.5317 - acc: 0.7999 10584/10584 [==============================] - 308s 29ms/step - loss: 0.3907 - acc: 0.8553 - val_loss: 0.1589 - val_acc: 0.9471 Epoch 3/30 10584/10584 [==============================] - 284s 27ms/step - loss: 0.1351 - acc: 0.9552 - val_loss: 0.0342 - val_acc: 0.9872 Epoch 4/30 10584/10584 [==============================] - 286s 27ms/step - loss: 0.0826 - acc: 0.9740 - val_loss: 0.0318 - val_acc: 0.9902 Epoch 5/30 10584/10584 [==============================] - 284s 27ms/step - loss: 0.0660 - acc: 0.9797 - val_loss: 0.0354 - val_acc: 0.9875 Epoch 6/30 10584/10584 [==============================] - 284s 27ms/step - loss: 0.0386 - acc: 0.9880 - val_loss: 0.0244 - val_acc: 0.9913 Epoch 7/30 10584/10584 [==============================] - 284s 27ms/step - loss: 0.0370 - acc: 0.9869 - val_loss: 0.0171 - val_acc: 0.9943 Epoch 8/30 10584/10584 [==============================] - 283s 27ms/step - loss: 0.0300 - acc: 0.9906 - val_loss: 0.0165 - val_acc: 0.9928 Epoch 9/30 10584/10584 [==============================] - 283s 27ms/step - loss: 0.0355 - acc: 0.9880 - val_loss: 0.0196 - val_acc: 0.9940 Epoch 10/30 10584/10584 [==============================] - 284s 27ms/step - loss: 0.0227 - acc: 0.9930 - val_loss: 0.0162 - val_acc: 0.9958 Epoch 11/30 10584/10584 [==============================] - 287s 27ms/step - loss: 0.0103 - acc: 0.9966 - val_loss: 0.0093 - val_acc: 0.9974 Epoch 12/30 10584/10584 [==============================] - 293s 28ms/step - loss: 0.0091 - acc: 0.9976 - val_loss: 0.0098 - val_acc: 0.9962 Epoch 13/30 10584/10584 [==============================] - 284s 27ms/step - loss: 0.0069 - acc: 0.9980 - val_loss: 0.0113 - val_acc: 0.9962 Epoch 14/30 10584/10584 [==============================] - 283s 27ms/step - loss: 0.0055 - acc: 0.9980 - val_loss: 0.0079 - val_acc: 0.9974 Epoch 15/30 10584/10584 [==============================] - 307s 29ms/step - loss: 0.0046 - acc: 0.9986 - val_loss: 0.0077 - val_acc: 0.9974 Epoch 16/30 10584/10584 [==============================] - 311s 29ms/step - loss: 0.0042 - acc: 0.9987 - val_loss: 0.0081 - val_acc: 0.9974 Epoch 17/30 10584/10584 [==============================] - 327s 31ms/step - loss: 0.0034 - acc: 0.9992 - val_loss: 0.0085 - val_acc: 0.9974 Epoch 18/30 10584/10584 [==============================] - 311s 29ms/step - loss: 0.0046 - acc: 0.9982 - val_loss: 0.0088 - val_acc: 0.9974 Epoch 19/30 10584/10584 [==============================] - 310s 29ms/step - loss: 0.0036 - acc: 0.9992 - val_loss: 0.0070 - val_acc: 0.9977 Epoch 20/30 10584/10584 [==============================] - 299s 28ms/step - loss: 0.0032 - acc: 0.9989 - val_loss: 0.0082 - val_acc: 0.9970 Epoch 21/30 10584/10584 [==============================] - 309s 29ms/step - loss: 0.0040 - acc: 0.9989 - val_loss: 0.0083 - val_acc: 0.9977 Epoch 22/30 10584/10584 [==============================] - 298s 28ms/step - loss: 0.0037 - acc: 0.9986 - val_loss: 0.0080 - val_acc: 0.9974 Epoch 23/30 10584/10584 [==============================] - 295s 28ms/step - loss: 0.0041 - acc: 0.9987 - val_loss: 0.0082 - val_acc: 0.9974 Epoch 24/30 10584/10584 [==============================] - 291s 28ms/step - loss: 0.0026 - acc: 0.9993 - val_loss: 0.0082 - val_acc: 0.9974 Epoch 25/30 2240/10584 [=====>........................] - ETA: 3:19 - los 2272/10584 [=====>........................] - ETA: 3:19 - los 2304/10584 [=====>........................] - ETA: 3:20 - loss: 0.0 2400/10584 [=====>........................] - ETA: 3:21 2432/10584 [=====>........................] - ETA: 3:21 - los 2464/10584 [=====>................10584/10584 [==============================] - 291s 27ms/step - loss: 0.0024 - acc: 0.9993 - val_loss: 0.0079 - val_acc: 0.9977 Epoch 26/30 10584/10584 [==============================] - 287s 27ms/step - loss: 0.0031 - acc: 0.9989 - val_loss: 0.0077 - val_acc: 0.9977 Epoch 27/30 10584/10584 [==============================] - 284s 27ms/step - loss: 0.0024 - acc: 0.9992 - val_loss: 0.0080 - val_acc: 0.9974 Epoch 28/30 10584/10584 [==============================] - 283s 27ms/step - loss: 0.0030 - acc: 0.9986 - val_loss: 0.0078 - val_acc: 0.9977 Epoch 29/30 10584/10584 [==============================] - 284s 27ms/step - loss: 0.0027 - acc: 0.9991 - val_loss: 0.0076 - val_acc: 0.9977 Epoch 30/30 10584/10584 [==============================] - 282s 27ms/step - loss: 0.0033 - acc: 0.9988 - val_loss: 0.0074 - val_acc: 0.9974 PREDICT

Test accuracy = 0