Why is history storing auc and val_auc with incrementing integers (auc_2, auc_4, ...)?

user_12853305 0
Viewed 744

I am beginner with keras and today I bumped into this sort of issue I don't know how to handle. The values for auc and val_auc are being stored in history with the first even integers, like auc, auc_2, auc_4, auc_6... and so on.

This is preventing me from managing and studying those values along my Kfold cross validation, as I cannot access history.history['auc'] value because there is not always such key 'auc'. Here is the code:

from tensorflow.keras.models import Sequential # pylint: disable= import-error
from tensorflow.keras.layers import Dense # pylint: disable= import-error
from tensorflow.keras import Input # pylint: disable= import-error
from sklearn.model_selection import StratifiedKFold
from keras.utils.vis_utils import plot_model
from keras.metrics import AUC, Accuracy # pylint: disable= import-error

BATCH_SIZE  = 32
EPOCHS      = 10
K           = 5
N_SAMPLE    = 1168
METRICS     = ['AUC', 'accuracy']

SAVE_PATH   = '../data/exp/final/submodels/'


def create_mlp(model_name, keyword, n_sample= N_SAMPLE, batch_size= BATCH_SIZE, epochs= EPOCHS):

    df = readCSV(n_sample)
    skf = StratifiedKFold(n_splits = K, random_state = 7, shuffle = True)

    for train_index, valid_index in skf.split(np.zeros(n_sample), df[['target']]):

        x_train, y_train, x_valid, y_valid = get_train_valid_dataset(keyword, df, train_index, valid_index)
        model = get_model(keyword)

        history = model.fit(
            x = x_train,
            y = y_train,
            validation_data = (x_valid, y_valid),
            epochs = epochs
        )

def get_train_valid_dataset(keyword, df, train_index, valid_index):
    aux = df[[c for c in columns[keyword]]]
    return aux.iloc[train_index].values, df['target'].iloc[train_index].values, aux.iloc[valid_index].values, df['target'].iloc[valid_index].values

def create_callbacks(model_name, save_path, fold_var):
    checkpoint = ModelCheckpoint(
        save_path + model_name + '_' +str(fold_var),
        monitor=CALLBACK_MONITOR, 
        verbose=1,
        save_best_only= True,
        save_weights_only= True,
        mode='max'
    )

    return [checkpoint]

In main.py I call create_mlp('model0', 'euler', n_sample=100), and the log is (only relevant lines):

Epoch 9/10
32/80 [===========>..................] - ETA: 0s - loss: 0.6931 - auc: 0.5000 - acc: 0.5625
Epoch 00009: val_auc did not improve from 0.50000
80/80 [==============================] - 0s 1ms/sample - loss: 0.6931 - auc: 0.5000 - acc: 0.5000 - val_loss: 0.6931 - val_auc: 0.5000 - val_acc: 0.5000
Epoch 10/10
32/80 [===========>..................] - ETA: 0s - loss: 0.6932 - auc: 0.5000 - acc: 0.4375
Epoch 00010: val_auc did not improve from 0.50000
80/80 [==============================] - 0s 1ms/sample - loss: 0.6931 - auc: 0.5000 - acc: 0.5000 - val_loss: 0.6931 - val_auc: 0.5000 - val_acc: 0.5000
Train on 80 samples, validate on 20 samples
Epoch 1/10
32/80 [===========>..................] - ETA: 0s - loss: 0.7644 - auc_2: 0.3075 - acc: 0.5000WARNING:tensorflow:Can save best model only with val_auc available, skipping.
80/80 [==============================] - 1s 10ms/sample - loss: 0.7246 - auc_2: 0.4563 - acc: 0.5250 - val_loss: 0.6072 - val_auc_2: 0.8250 - val_acc: 0.6500
Epoch 2/10
32/80 [===========>..................] - ETA: 0s - loss: 0.7046 - auc_2: 0.4766 - acc: 0.5000WARNING:tensorflow:Can save best model only with val_auc available, skipping.
80/80 [==============================] - 0s 1ms/sample - loss: 0.6511 - auc_2: 0.6322 - acc: 0.5625 - val_loss: 0.5899 - val_auc_2: 0.8000 - val_acc: 0.6000

Any help will be appreciated. I am using:

keras==2.3.1
tensorflow==1.14.0
keras tensorflow deep-learning python
3 Answers
user_78821900

In this line of code:

 for train_index, valid_index in skf.split(np.zeros(n_sample), df[['target']]):

What is actually happening is that you are running multiple training instances, in principle 5 as defaulted by sklearn.

Although you get different training and validation sets in :

 x_train, y_train, x_valid, y_valid = get_train_valid_dataset(keyword, df, train_index, valid_index)

When your run model.fit(),

   history = model.fit(
            x = x_train,
            y = y_train,
            validation_data = (x_valid, y_valid),
            epochs = epochs,
            callbacks=create_callbacks(keyword + '_' + model_name, SAVE_PATH, folder)
        )

You can see that the parameters for create_callbacks are static and do not change from one training instance to another. Keyword, model_name, SAVE_PATH and folder are arguments that remain constant during the 5 instances of your training.

Therefore, in TensorBoard, all the results at written at the same path.

You do not want to do that, you want each iteration to have its result written at different paths.

You have to change the logdir parameter, give it a unique identifier. In that situation, each training iteration will have written its results in separate locations, and thus the confusion will disappear.

user_61170170

I solved the issue by changing to tensorflow==2.1.0. Hope it can help anybody else.

user_128533050
Related