Saving entire model as TF SavedModel using Callback in TF 1.x?

user_14648214 0
Viewed 129

I would like to save my entire model every couple of epochs, including state of the optimizer, custom metrics etc. during training with keras model.fit_generator(). The goal is to be able to resume training from the exact state where it was left off.

The code I am working with is in TF 1.14. I know about the ModelCheckpoint callback, but it saves models in hdf5 format by default, which apparently can't deal properly with custom loss. I thought that probably the callback is just calling model.save() in the predefined interval during training, so is there a way to change file format passed to this call to SavedModel in versions earlier than TF 2.x, yielding my desired result?

keras tensorflow python
1 Answers

This code was tested with TF 1.15

import numpy as np
import keras

def custom_loss(y_true, y_pred):
    return y_pred*y_pred - y_true*y_true


m = keras.models.Sequential()
m.add(keras.layers.Dense(3, input_shape=(10,)))
m.compile('adam', loss=custom_loss)

x = np.random.randint(0,100,size=(200,10))
y = np.random.randint(0,100,size=(200,3))

mc = keras.callbacks.ModelCheckpoint('F:\\model.h5')
m.fit(x,y,callbacks=[mc])

Epoch 1/1
200/200 [==============================] - 0s 70us/step - loss: -622.8262

m = keras.models.load_model('F:\\model.h5', custom_objects={'custom_loss': custom_loss})

m.fit(x,y)

Epoch 1/1
200/200 [==============================] - 0s 244us/step - loss: -800.1645
user_34234470
Related