Tensorflow-Keras: Loading weights from checkpoint increases graph size significantly

user_10579155 0
Viewed 93

I am working with a rather large network (98 million parameters), I am using the Keras ModelCheckPoint callback to save my weights as follows, when I reload my saved weights using keras, I can see that the loading operation adds approximately 10 operations per layer in my graph. This results in a huge memory increase of my total network. Is this expected behavior? And if so, are there any known work arounds?

Details: I am using: tf.keras.callbacks.ModelCheckpoint with "save_weights_only=True" as argument to save the weights

The code for loading it is:

model.load_weights(path_to_existing_weights)

where model is a custom keras model.

I am using Tensorflow 1.14 and Keras 2.3.0

Anyone that has any ideas?

keras tensorflow python
1 Answers

This seems to me to be unexpected behavior but I can't see anything obvious that you are doing wrong. Are you sure there were no changes to your model between the time you saved the weights and the time you reloaded the weights? All I can suggest is try to do the same thing except this time in the callback change it to save the entire model. Then reload the model then check the graph. I also ran across this, doubt it is the problem but I would check it out

In order to save your Keras models as HDF5 files, e.g. via keras.callbacks.ModelCheckpoint, Keras uses the h5py Python package. It is a dependency of Keras and should be installed by default. If you are unsure if h5py is installed you can open a Python shell and load the module via import h5py If it imports without error it is installed, otherwise you can find detailed installation instructions here: http://docs.h5py.org/en/latest/build.html

Perhaps you might try reinstalling it.

user_107989170
Related