Tensorflow Slow after "Done running local_init_op"

Question

I have a tensorflow based code taken from github, which is very slow. It is even not printing (even after enabling DEBUG mode for tf.logging) what is happening after following is printed

INFO:tensorflow:Done running local_init_op. --> After this line code takes 20 minutes to execute INFO:tensorflow:prediction_loop marked as finished

Can someone please tell where to look at and optimize?
Following things are already checked:

model_fn : this gets executed before local_init_op
local files: checkpoint files are saved and loaded from local filesystem. So file transfer delay should not be the cause
warm_start_from: tried. Had no effect on prediction time

Estimator code:

estimator = tf.contrib.tpu.TPUEstimator(
      use_tpu=FLAGS.use_tpu,
      model_fn=model_fn,
      config=run_config,
      warm_start_from = tf.estimator.WarmStartSettings(
            ckpt_to_initialize_from='/content/ckpt',
        ),
      train_batch_size=FLAGS.train_batch_size,
      predict_batch_size=FLAGS.predict_batch_size)

Prediction Code:

results = estimator.predict(
          predict_input_fn, yield_single_examples=True, checkpoint_path='/content/ckpt/model.ckpt-10949')

Time taken when this code is executed:

results = list(results)

Tensorflow Slow after "Done running local_init_op"

0 Answers