OpenCV DNN forward() time inconsistency

Question

I wrote a simple code of forward()ing opencv DNN network in a while loop with 100ms delay.


m_net.setPreferableBackend(cv::dnn::DNN_BACKEND_CUDA);
m_net.setPreferableTarget(cv::dnn::DNN_TARGET_CUDA_FP16);

/*
...
*/

while(true)
{
    cv::Mat blob = cv::dnn::blobFromImage(in, 1 / 255.0, cv::Size(in.size[0]in.size[1]), cv::Scalar(), false, false, CV_32F);
    m_net.setInput(blob);
    m_net.forward();
    std::this_thread::sleep_for(std::chrono::milliseconds(100));
}

For the first few seconds, forward() takes around 7ms to process. However after some time(varies between 6 ~ 8 seconds), It suddenly starts to increase to 44ms. The GPU used here is disconnected from monitors thus does not get affected by displaying stuff on monitors. I ran Nsight System profiler and here's the result.

and here's the close up of the transition

The time it takes to do forward() has to be consistent since it will be used in a real time pipeline. What causes the issue and how can I resolve it?

Edit

More information about my setup:

OS: Windows 10
GPU: GeForce RTX 3090
OpenCV 4.5.3 with CUDA 11.1

I tried removing the delay and I observed no issue(forward() took consistent 7 ms). The problem seems to be the case only when the delay is present. Removing the delay cannot be a solution though, since the images the network has to process are acquired at the rate of 10 Hz and the delay simulates that.

OpenCV DNN forward() time inconsistency

0 Answers