Keras predict accuracy does not match to fit's result

user_1421239 0
Viewed 2684

I'm trying to build a binary classification model by TensorFlow 2.0 + Keras. Each target have 5 features and I want this model can predict whether the input data is belong to a.

However, the accuracy is totally different between fit() and predict(). The most weird thing is that I give the training data to the model for predict and the model does not return 1.

Construct the training data: (The a's features are marked to 1 and others are 0)

num_train = 50

data = {  # the content is fake, just for understanding the format
  'a': [(1, 2, 3, 4, 5), (2, 3, 4, 5, 6), ...],
  'b': [(10, 20, 30, 40, 50), (20, 30, 40, 50, 60), ...],
  ...
}

train_x = []
train_y = []

for name, features in data.items():
  for f in features[:num_train]:
    train_x.append(f)
    train_y.append(1 if name == 'a' else 0)

train_x = np.array(train_x)
train_y = np.array(train_y)

Here goes the model:

model = Sequential()
model.add(Dense(1, activation='sigmoid', input_dim=5))
model.compile(optimizer='sgd', loss='mse', metrics=['accuracy'])

And call model.fit():

model.fit(x=train_x, y=train_y, validation_split=0.2, batch_size=10, epochs=50)

After epoch 50:

Epoch 50/50
653/653 [==============================] - 0s 80us/sample - loss: 0.0745 - accuracy: 0.9234 - val_loss: 0.0192 - val_accuracy: 1.0000

At last, I use everyone's first 3 samples to predict:

for name, features in data.items():
  test_x = features[:3]
  print(name, np.around(model.predict(test_x), decimals=2))

The output:

a [[0.14] [0.14] [0.14]]
b [[0.14] [0.13] [0.13]]
c [[0.14] [0.14] [0.13]]
...

The complete data and source code is uploaded to Google Drive, please check the link.

keras tensorflow machine-learning python
1 Answers

After inspecting your source code, there are a few implementation issue:

  1. Training data and validation data are left randomized by Keras

During your training, 20% of the data is sampled to be the validation data, but you wouldn't know if the data sampled is balanced (i.e. same proportion of the classes in both training and validation data). In your case, because of the imbalanced, it is likely that the sampled training data were mostly from class 0, thus your model did not learn anything useful (thus the output is all the same 0.13 for all samples).

A better & more controlled approach is to split the data in a stratified manner before the training:

from sklearn.model_selection import train_test_split

num_train = 50

train_x = []
train_y = []

for name, features in data.items():
    for f in features[:num_train]:
        train_x.append(f)
        train_y.append(1 if name == 'a' else 0)

train_x = np.array(train_x)
train_y = np.array(train_y)

# Split your data, and stratify according to the target label `train_y`
# Set a random_state, so that the train-test split is reproducible

x_train, x_test, y_train, y_test = train_test_split(train_x, train_y, test_size=0.2, stratify=train_y, random_state=123)

and during train, you specify the validation_data instead of using validation_split:

model = Sequential()
model.add(Dense(1, activation='sigmoid', input_dim=5))
model.compile(optimizer='sgd', loss='binary_crossentropy', metrics=['accuracy'])
model.fit(x=x_train, y=y_train, 
          validation_data=(x_test, y_test), # Use this instead
          class_weight={0:1,1:17},  # See explanation in 2. Imbalanced class
          batch_size=10, epochs=500)
  1. Highly imbalanced class - Class 1 is 17 times less frequent than Class 0

Your class 1 a is 17 times lesser than class 0 (made up by the remaining). If you do not adjust for class weights, your model treat all samples equally, and by simply classifying everything as class 0 will give your model an accuracy of 94.4% (the remaining 5.6% are all from class 1, and were all classified wrongly by this naive model).

To account for class imbalance, a way is to set a higher loss for the minority class. In this example, I would set the classweight for class 1 to be 17 times more than class 0:

class_weight={0:1,1:17}

By doing so, you are telling the model that every sample from class 1 that is predicted wrongly results in 17 times more penalty than a mis-classified class 0. Thus the model is forced to pay more attention to class 1, despite it being a minority class.

  1. Thresholding not applied after getting the raw predictions.

After training (note that i increased the epochs to 500, and the model converged after about 200 epochs), do prediction on the test set which you obtained earlier:

preds = model.predict(x_test)

and you would get something like this:

[[0.33624142]
 [0.58196825]
 [0.5549609 ]
 [0.38138568]
 [0.45235538]
 [0.32419187]
 [0.37660158]
 [0.37013668]
 [0.5794893 ]
 [0.5611163 ]
 ......]

This is the raw output from neural network, which ranges from [0,1] because the last activation layer was sigmoid which squashes it to that range. In order to convert it to the class predictions that you need (class 0 or 1), a threshold needs to be applied. Normally, this threshold is set to 0.5, in which a prediction with output more than 0.5, means that the sample is likely to be from class 1, and otherwise for output less than 0.5.

Thus, you need to threshold the output using

threshold_output = np.where(preds > 0.5, 1, 0)

and you will get the actual class predictions:

[[0]
 [1]
 [1]
 [0]
 [0]
 [0]
 [0]
 [0]
 [1]
 [1]
 ...]

To get Training and Testing Accuracy

Now, to check for training and testing accuracy, you can use sklearn.metric directly, which saves you the hassle of calculating them manually:

from sklearn.metrics import accuracy_score

train_preds = np.where(model.predict(x_train) > 0.5, 1, 0)
test_preds = np.where(model.predict(x_test) > 0.5, 1, 0)

train_accuracy = accuracy_score(y_train, train_preds)
test_accuracy = accuracy_score(y_test, test_preds)

print(f'Train Accuracy : {train_accuracy:.4f}')
print(f'Test Accuracy  : {test_accuracy:.4f}')

which provides you:

Train Accuracy : 0.7443
Test Accuracy  : 0.7073

Hope this answers your question!

user_108411640
Related