I would like to load tensorflow model weights from a temporary file as part of a library I'm writing.
I'm using Tensorflow version 2.7.0
Consider the following tensorflow script meant to emulate this behavior and exposing the message. First I create a keras functional model, and train it on MNIST data. Then, I save weights to an .hdf5 file. I then create a temporary file, copy the weights into, and flush the file to disk. Then I use the temporary file to load the weights into a new TF model. Script here:
import tensorflow as tf
import tensorflow_datasets as tfds
import tempfile
import os
import shutil
# Load MNIST Datasets
print("Load MNIST Dataset")
mnist_datasets = tfds.load('mnist', as_supervised=True)
train_ds = mnist_datasets['train']
test_ds = mnist_datasets['test']
batch_size = 32
def normalize_img(image, label):
"""Normalizes images: `uint8` -> `float32`."""
return tf.cast(image, tf.float32) / 255., label
train_ds = train_ds.map(
normalize_img, num_parallel_calls=tf.data.AUTOTUNE)
train_ds = train_ds.cache()
train_ds = train_ds.shuffle(len(train_ds))
train_ds = train_ds.batch(batch_size)
test_ds = test_ds.map(
normalize_img, num_parallel_calls=tf.data.AUTOTUNE)
test_ds = test_ds.batch(batch_size)
test_ds = test_ds.cache()
test_ds = test_ds.prefetch(tf.data.AUTOTUNE)
# Define function to create functional keras model
def create_keras_model(layer_dims):
inp = tf.keras.layers.Input((28,28,1), name="Input_Name")
last_layer = inp
last_layer = tf.keras.layers.Reshape((28*28*1,), name="Input_Reshape")(last_layer)
for i in range(len(layer_dims)):
dim = layer_dims[i]
last_layer = tf.keras.layers.Dense(dim, name=f"Layer_{i}")(last_layer)
outp = tf.keras.layers.Dense(10, name="Output_Layer", activation='sigmoid')(last_layer)
return tf.keras.Model(inputs=inp, outputs=outp)
# Create and train initial model:
print("Create and train initial keras model")
mdl_1 = create_keras_model([32,32,32])
mdl_1.compile(
loss=tf.keras.losses.SparseCategoricalCrossentropy(),
metrics=tf.keras.metrics.SparseCategoricalAccuracy())
mdl_1.fit(train_ds, epochs=1)
# Create function to test models
def eval_on_test(mdl):
num_matches = 0
total_examples = 0
for X, Y in test_ds:
# Eval model
Y_eval = tf.argmax(mdl(X),axis=1).numpy()
# Count
total_examples += len(Y_eval)
num_matches += (Y.numpy() == Y_eval).sum()
return num_matches/total_examples
print("Initial model performance")
print(eval_on_test(mdl_1))
# Save weights to disk
mdl_weight_file = 'weights.hdf5'
mdl_1.save_weights(mdl_weight_file)
# Create new model and load from a temporary file
mdl_2 = create_keras_model([32,32,32])
with tempfile.NamedTemporaryFile('w+b') as temp_f:
# Copy content to temporary file
print("Copy weights to temp file")
with open(mdl_weight_file, 'rb') as f:
temp_f.write(f.read())
# Flush to disk
temp_f.flush()
print("Load weights from temp file")
# Load weights from temp file
mdl_2.load_weights(temp_f.name)
print("temp file gone")
print("Loaded model performance")
print(eval_on_test(mdl_2))
When I run this script I get the following output:
python TempFileKerasWeightsLoadIssue.py
Load MNIST Dataset
2022-01-31 10:48:45.339270: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2022-01-31 10:48:45.339318: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (hal-login2.ncsa.illinois.edu): /proc/driver/nvidia/version does not exist
Create and train initial keras model
1875/1875 [==============================] - 6s 2ms/step - loss: 0.3802 - sparse_categorical_accuracy: 0.8922
Initial model performance
0.9137
Copy weights to temp file
Load weights from temp file
2022-01-31 10:48:52.655314: W tensorflow/core/util/tensor_slice_reader.cc:96] Could not open /tmp/tmpg3kjnf7s: DATA_LOSS: not an sstable (bad magic number): perhaps your file is in a different file format and you need to use a different restore operator?
temp file gone
Loaded model performance
0.9137
Now, the loaded model performance is the same as the trained model, so loading the weights seems to have been successful, however Tensorflow is still producing this annoying warning message indicating that the file wasn't there! It seems Tensorflow is trying to read/write to the weights file after the load is completed and the temp file is destroyed in my script. How can I ensure that Tensorflow is done with this file before removing the temporary file? Thus quashing the warning message.