I wish to add functionality to my code such that if I desire to terminate the code at any point it will safely terminate training and save the variables. Although I've tried searching for a better solution, I think catching a KeyboardInterrupt exception would be my best bet.
Would it, however, be safe? More specifically, would the following code work:
with tf.Session() as sess
try:
for i in range(FLAGS.max_steps):
sess.run(train_op, feed_dict=some_feed_dictionary)
# Some other summary writing and evaluative operations
except KeyboardInterrupt:
print("Manual interrupt occurred.")
print('Done training for {} steps'.format(global_steps))
save_path = saver.save(sess, 'Standard CNN', global_step=global_steps, write_meta_graph=False)
Or is it unsafe and can result in corrupted save files considering the Keyboard Interrupt is free to occur in the middle of any tensorflow operation? Is there an adequate way of doing this?