I am using the TensorFlow (1.2, GPU) implementation of Faster R-CNN (here). When I run a VGG16 network (trained on PASCAL VOC07) on some sample images using the demo.py code, the running time is weird: it changes a lot even if I supply the same images.
Side note: all the input images are resized so that shorter side is 600 pixels and longer side is max 1000 pixels; hence all the input images have roughly the same size.
GPU: GTX 1060 6GB
Using tfconfig.gpu_options.allow_growth=True
Image resolutions, 000456.jpg: 500x375; 000188.png: 1224x370
Demo for data/demo/000456.jpg
Detection took 2.187s for 300 object proposals
Demo for data/demo/000456.jpg
Detection took 0.141s for 300 object proposals
Demo for data/demo/000188.png
Detection took 1.321s for 300 object proposals
Demo for data/demo/000188.png
Detection took 0.112s for 300 object proposals
Changing the order of the images:
Demo for data/demo/000188.png
Detection took 1.760s for 300 object proposals
Demo for data/demo/000188.png
Detection took 0.109s for 300 object proposals
Demo for data/demo/000456.jpg
Detection took 1.775s for 300 object proposals
Demo for data/demo/000456.jpg
Detection took 0.142s for 300 object proposals
Using tfconfig.gpu_options.per_process_gpu_memory_fraction = 0.8
Demo for data/demo/000188.png
Detection took 1.699s for 300 object proposals
Demo for data/demo/000188.png
Detection took 0.114s for 300 object proposals
Demo for data/demo/000456.jpg
Detection took 1.782s for 300 object proposals
Demo for data/demo/000456.jpg
Detection took 0.142s for 300 object proposals
It seems, when the image size changes, either from small to large or large to small, the first image takes a long time to process, while the following images of similar sizes take much shorter (around 0.1 seconds). Similar behavior when I supply all different images.
I could not think of an explanation for this weird behavior. It seems to be somehow related to memory allocation/deallocation, but how? Do you know what happens and how to avoid this behavior, i.e., the first image may take a long time to process (memory allocation, cache, etc.), but all the subsequent images should take the normal/shorter amount of time (I think).
EDIT: The following shows the running times on PASCAL VOC 2007 object detection test set (with tfconfig.gpu_options.allow_growth=True). In this case, the inference time gradually decreases from 2.3 seconds to 0.25 seconds, although the image sizes are roughly the same.
im_detect: 1/4952 2.326s 0.001s
im_detect: 2/4952 2.153s 0.001s
im_detect: 3/4952 2.047s 0.001s
im_detect: 4/4952 1.971s 0.001s
im_detect: 5/4952 1.606s 0.001s
im_detect: 6/4952 1.363s 0.001s
im_detect: 7/4952 1.433s 0.001s
im_detect: 8/4952 1.507s 0.001s
im_detect: 9/4952 1.356s 0.001s
im_detect: 10/4952 1.424s 0.001s
im_detect: 11/4952 1.307s 0.001s
im_detect: 12/4952 1.210s 0.001s
im_detect: 13/4952 1.198s 0.001s
im_detect: 14/4952 1.123s 0.001s
...
im_detect: 60/4952 0.733s 0.001s
im_detect: 61/4952 0.723s 0.001s
im_detect: 62/4952 0.713s 0.001s
im_detect: 63/4952 0.704s 0.001s
im_detect: 64/4952 0.696s 0.001s
im_detect: 65/4952 0.687s 0.001s
...
im_detect: 250/4952 0.463s 0.001s
im_detect: 251/4952 0.462s 0.001s
im_detect: 252/4952 0.468s 0.001s
im_detect: 253/4952 0.466s 0.001s
im_detect: 254/4952 0.470s 0.001s
im_detect: 255/4952 0.469s 0.001s
...
im_detect: 4948/4952 0.245s 0.001s
im_detect: 4949/4952 0.245s 0.001s
im_detect: 4950/4952 0.245s 0.001s
im_detect: 4951/4952 0.245s 0.001s
im_detect: 4952/4952 0.245s 0.001s