Loading from Keras checkpoint

user_10400605 0
Viewed 9512

I am training a model in Keras where I saved everythinig using following codes.

filepath = "project_model.hdh5"

checkpoint = ModelCheckpoint("project_model.hdf5", monitor='loss', verbose=1,
    save_best_only=False, mode='auto', period=1)

Then I used the following codes to run the training.

for _ in range(20):
    for j in range(len(mfcc_data_padded_transposed[j])):
        batch_input=[mfcc_data_padded_transposed[j]]
        batch_input = np.array(batch_input)
        batch_input = batch_input/np.max(batch_input)
        batch_output = [y_labels_mfcc[j]]
        batch_output = np.array(batch_output)
        input_lengths2 = input_lengths_mfcc[j]
        label_lengths2 = label_lengths_mfcc[j]
        input_lengths2 = np.array(input_lengths2)
        label_lengths2 = np.array(label_lengths2)
        inputs = {'the_input': batch_input,
         'the_labels': batch_output,
         'input_length': input_lengths2,
         'label_length': label_lengths2}
        outputs = {'ctc': np.zeros([1])} 
        model.fit(inputs, outputs, epochs=1, verbose =1, callbacks=[checkpoint])

I did the above to experiment with the checkpoint because I am not sure whether I am using that correctly.

Now, this training is done with a learning rate of .001. Now after running the training loop for a while if I decide to change the learning rate to say .002, would I have to run all the codes that are related to the models (the model structure, then the optimization, etc)? And say I did that, how do I load from the previous state from when I stopped the training? Other question is, if I restart the PC, and run the jupyter cell with checkpoint codes that I shared here earlier, would that replace the previously saved file? What is an ideal way to load the saved files and weights and resume training from there? Reason that I am asking is because it seems that when I followed the Keras documentation, it just starts from the start.

keras tensorflow python
1 Answers

Now after running the training loop for a while if I decide to change the learning rate to say .002, would I have to run all the codes that are related to the models (the model structure, then the optimization, etc)?

You can update the learning rate either during training or after loading your model.

Remember that the learning rate does not belong to the model architecture, it belongs to the optimizer (which is assigned during model compilation). The learning rate is a hyperparameter that regulates the magnitude of weight update during gradient descent (represented as alpha below):

enter image description here

So after initial training, you can load your (saved) model, update the optimizer with a new learning rate (and perhaps assign a custom object to the compiler), and continue training. Keep in mind that changing the optimizer itself after having trained your model for a long time can produce poor accuracy results, as your model now has to re-calibrate to the new optimizer's weight calculations.

how do I load from the previous state from when I stopped the training?

In Keras you have the choice of saving/loading the whole model (which includes the architecture, weights, optimizer state; or just the weights; or just the architecture (source).

To save/load whole model:

from keras.models import load_model

model.save('my_model.h5')
model = load_model('my_model.h5')

To only save/load model weights:

model.save_weights('my_model_weights.h5')
model.load_weights('my_model_weights.h5')

You can also assign a custom object during model loading:

model = load_model(filepath, custom_objects={'loss': custom_loss})

Other question is, if I restart the PC, and run the jupyter cell with checkpoint codes that I shared here earlier, would that replace the previously saved file?

Depends on the filepath used in the checkpoint: "if filepath is weights.{epoch:02d}-{val_loss:.2f}.hdf5, then the model checkpoints will be saved with the epoch number and the validation loss in the filename". So, if you're using unique formatting for the filepath, then you can avoid overwriting previously saved models. source

What is an ideal way to load the saved files and weights and resume training from there?

Example:

# Define model
model = keras.models.Sequential()

model.add(L.InputLayer([None],dtype='int32'))
model.add(L.Embedding(len(all_words),50))
model.add(keras.layers.Bidirectional(L.SimpleRNN(5,return_sequences=True)))

# Define softmax layer for every time step (hence TimeDistributed layer)
stepwise_dense = L.Dense(len(all_words),activation='softmax')
stepwise_dense = L.TimeDistributed(stepwise_dense)
model.add(stepwise_dense)

import keras.backend as K

# compile model with adam optimizer
model.compile('adam','categorical_crossentropy')

# print learning rate
print(f"Model learning rate is: {K.get_value(model.optimizer.lr):.3f}")

# train model
model.fit_generator(generate_batches(train_data), len(train_data)/BATCH_SIZE,
                    callbacks=[EvaluateAccuracy()], epochs=1)

# save model (weights, architecture, optimizer state)
model.save('my_model.h5')

# delete existing model
del model

Results

Model learning rate is: 0.001
Epoch 1/1
1341/1343 [============================>.] - ETA: 0s - loss: 0.4288
Measuring validation accuracy...
Validation accuracy: 0.93138
from keras.models import load_model

# create new adam optimizer with le-04 learning rate (previous: 1e-03)
adam = keras.optimizers.Adam(lr=1e-4)

# load model
model = load_model('my_model.h5', compile=False)

# compile model and print new learning rate
model.compile(adam, 'categorical_crossentropy')
print(f"Model learning rate is: {K.get_value(model.optimizer.lr):.4f}")

# train model for 3 more epochs with new learning rate
print("Training model: ")
model.fit_generator(generate_batches(train_data),len(train_data)/BATCH_SIZE,
                    callbacks=[EvaluateAccuracy()], epochs=3,)

Results:

Model learning rate is: 0.0001
Training model: 
Epoch 1/3
1342/1343 [============================>.] - ETA: 0s - loss: 0.0885
Measuring validation accuracy...
Validation accuracy: 0.93568

1344/1343 [==============================] - 41s - loss: 0.0885    
Epoch 2/3

1342/1343 [============================>.] - ETA: 0s - loss: 0.0768
Measuring validation accuracy...
Validation accuracy: 0.93925

1344/1343 [==============================] - 39s - loss: 0.0768    
Epoch 3/3
1343/1343 [============================>.] - ETA: 0s - loss: 0.0701
Measuring validation accuracy...
Validation accuracy: 0.94180

More info at Keras FAQ for specific cases.

user_80725460
Related