In the sample pipeline config file of TensorFlow object detection, there is this snippet:
eval_config: {
num_examples: 2000
# Note: The below line limits the evaluation process to 10 evaluations.
# Remove the below line to evaluate indefinitely.
max_evals: 10
}
Does "num_examples" mean each evaluation run uses the same first 2000 images, or it treats the test set as a circular buffer and uses different 2000 images each time?
Actually this means only the same top num_examples samples in your evaluation dataset will be used in each run of evaluation.
TL;DR Circular buffer if enough num_epochs and no shuffle
I believe it works in "collaboration" with the input reader config. If in the eval_input_reader you set num_epochs to 1, then it will process the first 2000 images from the input queue, provided the shuffle = false, otherwise some random 2000 images. If you don't have 2000 images, it will probably fail, as the queue is emptied.