All runs are not visible on TensorBoard

user_2902280 0
Viewed 7141

I am using tensorboard to visualize three runs. I have a folder, logs, which contains three files: 

2016-03-18_22-11-12
2016-03-18_22-11-27
2016-03-18_22-23-46

when I run tensorboard --logdir . (from logs), only 2016-03-18_22-23-46 is visible:blu

And if I delete 2016-03-18_22-23-46 from logs and restart tensorboard, then only 2016-03-18_22-11-27 is visible. Any idea of what's happening here?

Edit: the log files are (to my surprise), quite big: here is the result of du -h:

1,1G    ./2016-03-18_22-23-46
925M    ./2016-03-18_22-11-12
934M    ./2016-03-18_22-11-27
2,9G    .

EDIT: The above run structure can be obtained via logging and checkpointing into subdirectories of the tensorboard log-dir:

run_time = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
FLAGS.train_dir = '/datalab/tf_runs/' + run_time #Directory to put the training data.
summary_writer = tf.train.SummaryWriter(FLAGS.train_dir, sess.graph_def)
tensorboard tensorflow
3 Answers

Another solution is to use the --max_reload_threads option as follows:

tensorboard --logdir=runs --max_reload_threads 4

The no. of threads could be determined based on the fact that one thread can parse one run at a time.

user_102802250

TL;DR: Close any currently running jupyter notebook / python file that has Tensorboard callbacks.

I had the same problem, only one run was visualized in Tensorboard and multiple weren't listed. I waited and refreshed some time, as @etarion pointed out in his answer, but that didn't make the other runs appear.

Then I found out what was happening: I had just finished running training in my notebook file. Then I wanted to open all previous runs in Tensorboard, and the only one appearing was the current one I had just trained. I first had to close the notebook file and then running Tensorboard showed all runs!

For some reason the notebook still being active led Tensorboard only take this run.

user_120971910
Related