(Novice question; sorry in advance.)
I have a pretrained (BigGAN) tensorflow model that another user developed to create images, and I would like to use it to generate new images. (If it helps to simplify the process, I do not need to train the model further; I just want to sample from the last checkpoint.) The model consists of the following four files:
model.ckpt-607250.data-00000-of-00001
model.ckpt-607250.index
model.ckpt-607250.meta
operative_config-603500.gin
I've been trying to use Google Colab (with TPU processing enabled) to restore the model using the following code:
import tensorflow as tf
import os
tf.compat.v1.enable_eager_execution()
model_path = "/content/drive/My Drive/biggan/"
resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)
tf.compat.v1.disable_eager_execution()
model_path = "/content/drive/My Drive/biggan/"
saver = tf.compat.v1.train.import_meta_graph(model_path+'model.ckpt-607250.meta')
with tf.device('/TPU:0'):
with tf.compat.v1.Session() as sess:
saver.restore(sess, model_path+'model.ckpt-607250')
However, this triggers the following error:
InvalidArgumentError Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1364 try:
-> 1365 return fn(*args)
1366 except errors.OpError as e:
9 frames
InvalidArgumentError: No OpKernel was registered to support Op 'TPUReplicatedInput' used by {{node input0}} with these attrs: [is_mirrored_variable=false, index=-1, T=DT_INT32, N=128, is_packed=false]
Registered devices: [CPU, XLA_CPU]
Registered kernels:
<no registered kernels>
[[input0]]
During handling of the above exception, another exception occurred:
InvalidArgumentError Traceback (most recent call last)
During handling of the above exception, another exception occurred:
InvalidArgumentError Traceback (most recent call last)
/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/saver.py in restore(self, sess, save_path)
1333 # We add a more reasonable error message here to help users (b/110263146)
1334 raise _wrap_restore_error_with_msg(
-> 1335 err, "a mismatch between the current graph and the graph")
1336
1337 @staticmethod
InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
No OpKernel was registered to support Op 'TPUReplicatedInput' used by node input0 (defined at <ipython-input-2-009a7264c288>:11) with these attrs: [is_mirrored_variable=false, index=-1, T=DT_INT32, N=128, is_packed=false]
Registered devices: [CPU, XLA_CPU]
Registered kernels:
<no registered kernels>
[[input0]]
No matter what I do, I keep running into the same error with TPUReplicatedInput, and I'm not sure how to resolve it. I've been struggling with this for a few weeks, so any help that you can offer would be greatly appreciated.