Constant validation accuracy in binary classification using ResNet50

user_11258124 0
Viewed 668

I am training a deep learning network to classify thyroid nodules (binary classification as 0 or 1). Therefore, I am realizing that my model is with constant validation accuracy. I tried everything, changed the learning rate, some callbacks if the loss doesn't increse. So, I need help, some ideas to deal with it. I will give you a example with 10 epochs, but, even if I change to 100 epochs, it still with the same behavior. Here is the architecture's network code:

callbacks = [
    EarlyStopping(patience=20, verbose=1),
    ReduceLROnPlateau(factor=0.1, patience=5, min_lr=0.0001, verbose=1),
]

epochs=10
model = Sequential()
model.add(tf.keras.applications.ResNet50( include_top=False, pooling='avg', weights='imagenet'))
model.add(BatchNormalization())
model.add(Dropout(0.4))
model.add(Dense(2, activation='sigmoid'))
model.layers[0].trainable = False
weights=class_weight.compute_class_weight('balanced', np.unique(y_train), y_train)
opt =Adam(learning_rate=0.001)
model.compile(loss='categorical_crossentropy', optimizer=opt, metrics=['accuracy'])
tf.random.set_seed(2)
rede=model.fit(X_train, y_train_2d, epochs=epochs, batch_size=18, class_weight=weights, callbacks=callbacks,\
               validation_data=(X_valid, y_valid_2d))

Train on 360 samples, validate on 46 samples
Epoch 1/10
342/360 [===========================>..] - ETA: 1s - loss: 0.7582 - accuracy: 0.5673
Epoch 00001: val_loss did not improve from 0.16301
360/360 [==============================] - 44s 123ms/sample - loss: 0.7706 - accuracy: 0.5583 - val_loss: 0.5528 - val_accuracy: 0.8043
Epoch 2/10
342/360 [===========================>..] - ETA: 1s - loss: 0.5604 - accuracy: 0.7135
Epoch 00002: val_loss did not improve from 0.16301
360/360 [==============================] - 41s 114ms/sample - loss: 0.5525 - accuracy: 0.7222 - val_loss: 0.5047 - val_accuracy: 0.8043
Epoch 3/10
342/360 [===========================>..] - ETA: 1s - loss: 0.5056 - accuracy: 0.7485
Epoch 00003: val_loss did not improve from 0.16301
360/360 [==============================] - 41s 114ms/sample - loss: 0.4993 - accuracy: 0.7500 - val_loss: 0.4946 - val_accuracy: 0.8043
Epoch 4/10
342/360 [===========================>..] - ETA: 1s - loss: 0.5004 - accuracy: 0.7661
Epoch 00004: val_loss did not improve from 0.16301
360/360 [==============================] - 41s 114ms/sample - loss: 0.5019 - accuracy: 0.7667 - val_loss: 0.4942 - val_accuracy: 0.8043
Epoch 5/10
342/360 [===========================>..] - ETA: 1s - loss: 0.4967 - accuracy: 0.8070
Epoch 00005: val_loss did not improve from 0.16301
360/360 [==============================] - 41s 114ms/sample - loss: 0.4897 - accuracy: 0.8083 - val_loss: 0.4952 - val_accuracy: 0.8043
Epoch 6/10
342/360 [===========================>..] - ETA: 1s - loss: 0.4615 - accuracy: 0.7427
Epoch 00006: val_loss did not improve from 0.16301
360/360 [==============================] - 41s 114ms/sample - loss: 0.4750 - accuracy: 0.7361 - val_loss: 0.4985 - val_accuracy: 0.8043
Epoch 7/10
342/360 [===========================>..] - ETA: 1s - loss: 0.4119 - accuracy: 0.7924
Epoch 00007: val_loss did not improve from 0.16301
360/360 [==============================] - 40s 111ms/sample - loss: 0.4164 - accuracy: 0.7944 - val_loss: 0.4953 - val_accuracy: 0.8043
Epoch 8/10
342/360 [===========================>..] - ETA: 0s - loss: 0.4182 - accuracy: 0.8158
Epoch 00008: val_loss did not improve from 0.16301
360/360 [==============================] - 20s 55ms/sample - loss: 0.4155 - accuracy: 0.8194 - val_loss: 0.4957 - val_accuracy: 0.8043
Epoch 9/10
342/360 [===========================>..] - ETA: 0s - loss: 0.3892 - accuracy: 0.8304
Epoch 00009: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.

Epoch 00009: val_loss did not improve from 0.16301
360/360 [==============================] - 21s 57ms/sample - loss: 0.3853 - accuracy: 0.8278 - val_loss: 0.4997 - val_accuracy: 0.8043
Epoch 10/10
342/360 [===========================>..] - ETA: 0s - loss: 0.3395 - accuracy: 0.8333
Epoch 00010: val_loss did not improve from 0.16301
360/360 [==============================] - 21s 58ms/sample - loss: 0.3375 - accuracy: 0.8333 - val_loss: 0.5003 - val_accuracy: 0.8043
resnet keras tensorflow deep-learning
3 Answers

For binary classification, you should change output dense layer to 1 unit and loss to binary_crossentropy.

model.add(Dense(1, activation='sigmoid'))
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])

Also make sure your target arrays (y_train,y_valid) have correct shape and data. They should contain binary labels, 0 or 1

user_134544540

@AlexM4 is right that, for binary classification, you should use a single output unit with a sigmoid classification.

What's also worth pointing out is that, for multi-class classification, you shouldn't use a sigmoid activation but instead softmax, which produces a probability distribution over the possible classes (sigmoid could theoretically produce a probability of 1.0 for all classes, which makes no sense). That may be impeding your network's performance here.

Fixing the larger issue (moving to a single output unit) would obviate the need to fix the issue I just described, but it's worth keeping in mind for future experiments.

user_66964080

I tried to change the output dense to 1, instead of 2. The accuracy still with the same behavior. 

epochs=10
model = Sequential()
model.add(tf.keras.applications.ResNet50( include_top=False, pooling='avg', weights='imagenet'))
model.add(BatchNormalization())
model.add(Dropout(0.4))
model.add(Dense(1, activation='sigmoid'))
model.layers[0].trainable = False
weights=class_weight.compute_class_weight('balanced', np.unique(y_train), y_train)
opt =Adam(learning_rate=0.001)
model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
tf.random.set_seed(2)
rede=model.fit(X_train, y_train, epochs=epochs, batch_size=18, class_weight=weights, callbacks=callbacks,\
               validation_data=(X_valid, y_valid))
Train on 360 samples, validate on 46 samples
Epoch 1/10
360/360 [==============================] - 17s 48ms/sample - loss: 0.8778 - accuracy: 0.5639 - val_loss: 0.3967 - val_accuracy: 0.8696
Epoch 2/10
360/360 [==============================] - 9s 24ms/sample - loss: 0.6527 - accuracy: 0.7028 - val_loss: 0.3973 - val_accuracy: 0.8696
Epoch 3/10
360/360 [==============================] - 9s 26ms/sample - loss: 0.6054 - accuracy: 0.7222 - val_loss: 0.4171 - val_accuracy: 0.8696
Epoch 4/10
360/360 [==============================] - 10s 28ms/sample - loss: 0.5388 - accuracy: 0.7500 - val_loss: 0.4211 - val_accuracy: 0.8696
Epoch 5/10
360/360 [==============================] - 11s 30ms/sample - loss: 0.4830 - accuracy: 0.7694 - val_loss: 0.4367 - val_accuracy: 0.8696
Epoch 6/10
342/360 [===========================>..] - ETA: 0s - loss: 0.4986 - accuracy: 0.8070
Epoch 00006: ReduceLROnPlateau reducing learning rate to 0.00010000000474974513.
360/360 [==============================] - 11s 29ms/sample - loss: 0.4952 - accuracy: 0.8056 - val_loss: 0.4576 - val_accuracy: 0.8696
Epoch 7/10
360/360 [==============================] - 11s 30ms/sample - loss: 0.4257 - accuracy: 0.8306 - val_loss: 0.4661 - val_accuracy: 0.8696
Epoch 8/10
360/360 [==============================] - 17s 46ms/sample - loss: 0.4054 - accuracy: 0.8250 - val_loss: 0.4722 - val_accuracy: 0.8696
Epoch 9/10
360/360 [==============================] - 17s 48ms/sample - loss: 0.4484 - accuracy: 0.7917 - val_loss: 0.4761 - val_accuracy: 0.8696
Epoch 10/10
360/360 [==============================] - 16s 44ms/sample - loss: 0.3844 - accuracy: 0.8472 - val_loss: 0.4785 - val_accuracy: 0.8696
user_112581240
Related