I'm using the new tensorflow object detection API on my own data. It works very well, but I'm a bit disappointed that some statistics shown in Tensorboard are only there for the training set, not the eval, and inversely. For instance, I think it would be great to get the global_step/sec and the losses during evaluation, and also the Precision and Performance metrics for the training step (without evaluating manually on it).
Is there any simple way to do that ?
I'm training and testing with the scripts provided by the API, with pretty standard configuration files, with both SSD and Faster-RCNN:
python tensorflow_models_dir/object_detection/train.py --logtostderr --pipeline_config_path=../models/SSD_v1/config_v1.config --train_dir=../models/SSD_v1/train
python tensorflow_models_dir/object_detection/eval.py --logtostderr --pipeline_config_path=../models/SSD_v1/config_v1.config --checkpoint_dir=../models/SSD_v1/train --eval_dir=../models/SSD_v1/eval
So far I've tried adding the summaries from trainer.py to the tensor_dict in evaluator.py, but it failed at runtime. To do this, I've added the following lines to _extract_prediction_tensors in evaluator.py:
# Gather initial summaries.
summaries = set(tf.get_collection(tf.GraphKeys.SUMMARIES))
global_summaries = set([])
# Add summaries.
for model_var in slim.get_model_variables():
global_summaries.add(tf.summary.histogram(model_var.op.name, model_var))
for loss_tensor in tf.losses.get_losses():
global_summaries.add(tf.summary.scalar(loss_tensor.op.name, loss_tensor))
# global_summaries.add(
# tf.summary.scalar('TotalLoss', tf.losses.get_total_loss())) # Crashes
# Add the summaries from the first clone. These contain the summaries
# created by model_fn and either optimize_clones() or _gather_clone_loss().
summaries |= set(tf.get_collection(tf.GraphKeys.SUMMARIES))
summaries |= global_summaries
# Merge all summaries together.
summary_op = tf.summary.merge(list(summaries), name='summary_op')
tensor_dict['summary_op'] = summary_op