Restoring queue state in Tensorflow from checkpoint

Question

Context: I am training a model using an Estimator. Without extraneous details, I am using queues to read in a series of input images, which are batched and manipulated using an input function which I am call "read_pics_batch":

with tf.Session() as sess:
   sess.run(tf.global_variables_initializer())
   sess.run(tf.local_variables_initializer())
   coord = tf.train.Coordinator()
   threads = tf.train.start_queue_runners(sess=sess, coord=coord)

   keypoint_regression.fit(
       input_fn = lambda: inp_mod.read_pics_batch(names_train, \
       joint_annopoints_train,num_in_batch,max_num_epochs,'TRAIN'),
       steps= max_steps_per_epoch*max_num_epochs, # max number of steps
       monitors=[logging_hook])

   coord.request_stop()
   coord.join(threads)

The input function has the following form, where I am also randomising the input file order:

def read_pics_batch(names_list,joint_list,batch_size,max_num_epochs,task):

    names_tensor = tf.convert_to_tensor(names_list, dtype=tf.string)
    joint_total_tensor = tf.convert_to_tensor(joint_list, dtype=tf.int32)

    min_after_dequeue = 100
    capacity = min_after_dequeue + 3 * batch_size

    file_pattern = [("...")]

    examples = graph_io.read_keyed_batch_examples(file_pattern, batch_size, \
        reader = tf.WholeFileReader, randomize_input = True, \
        parse_fn = example_to_standard_pic, \
        num_epochs = max_num_epochs, queue_capacity = capacity)

My questions are as follows:

1) Is there any way to restore the queue state from a checkpoint, like any other variable? If "randomize_input" from read_keyed_batch_examples would be set to False, than at each restart of the training_op I would read the same input files over and over, which is clearly not what I want.

2) If randomize_input = True, how exactly does the queue decide which files to enqueue? I see two possible options and I am unsure which is correct:

it selects a short-list of size "capacity" (from the full-list given by all the filenames defined by "file_pattern") and then randomises the order of the names in this short-list
it randomises the names in the full-list first, and then creates a short-list of size "capacity" out of this

If the second case applies, I don't believe that I would actually need to restore the queue state, since I would in principle read different files every time, but if the first case applies, I would still be reading the same few files over and over, just in a different order.

Thank you for your time!

Restoring queue state in Tensorflow from checkpoint

0 Answers