Why opencv videowriter is so slow?

Question

Hi stackoverflow community, I have a tricky problem and I need your help to understand what is going on here. My program captures frames from a video grabber card (Blackmagic) which just works fine so far, at the same time I display the captured images with opencv (cv::imshow) which works good as well (But pretty cpu wasting). The captured images are supposed to be stored on the disk as well, for this I put the captured Frames (cv::Mat) on a stack, to finally write them async with opencv:

cv::VideoWriter videoWriter(path, cv::CAP_FFMPEG, fourcc, fps, *size);
videoWriter.set(cv::VIDEOWRITER_PROP_QUALITY, 100);

int id = metaDataWriter.insertNow(path);

while (this->isRunning) {

    while (!this->stackFrames.empty()) {

        cv:Mat m = this->stackFrames.pop();

        videoWriter << m;
    }
    
}

videoWriter.release();

This code is running in an additional thread and will be stopped from outside. The code is working so far, but it is sometimes pretty slow, which means my stack size increases and my system runs out of ram and get killed by the OS.

Currently it is running on my developing system:

Ubuntu 18.04.05
OpenCV 4.4.0 compiled with Cuda
Intel i7 10. generation 32GB RAM, GPU Nvidia p620, M.2 SSD

Depending on the codec (fourcc) this produces a high CPU load. So far I used mainly "MJPG", "x264". Sometimes even MJPG turns one core of the CPU to 100% load, and my stack raises until the programs run out of run. After a restart, sometimes, this problem is fixed, and it seems the load is distributed over all cores.

Regarding to the Intel Doc. for my CPU, it has integrated hardware encoding/decoding for several codecs. But I guess opencv is not using them. Opencv even uses its own ffmpeg and not the one of my system. Here is my build command of opencv:

cmake -D CMAKE_BUILD_TYPE=RELEASE \
-D CMAKE_INSTALL_PREFIX=/usr/local \
-D WITH_TBB=ON \
-D WITH_CUDA=ON \
-D BUILD_opencv_cudacodec=OFF \
-D ENABLE_FAST_MATH=1 \
-D CUDA_FAST_MATH=1 \
-D WITH_CUBLAS=1 \
-D WITH_V4L=ON \
-D WITH_QT=OFF \
-D WITH_OPENGL=ON \
-D WITH_GSTREAMER=ON \
-D OPENCV_GENERATE_PKGCONFIG=ON \
-D OPENCV_ENABLE_NONFREE=ON \
-D WITH_FFMPEG=1 \
-D OPENCV_EXTRA_MODULES_PATH=../../opencv_contrib/modules \
-D WITH_CUDNN=ON \
-D OPENCV_DNN_CUDA=ON \
-D CUDA_ARCH_BIN=6.1 ..

I just started development with linux and C++, before I was working with Java/Maven, so the use of cmake is still a work in progress, pls go easy on me.

Basically my question is, how can I make the video encoding/writing faster, use the hardware acceleration at best? Or if you think there is something else fishy, pls let me know.

BR Michael

user_2267367 · Answer

-------- old - look up answer on bottom --------

Thank @Micka for the many advises, I found the right thing on the way.

Using cudacodec::VideoWriter is not that easy, after compiling I was not able to use it because of this error, and even if I can make it run, the deployment PC does not have a nvidia GPU.

Since I am going to use PCs with AMD CPUs as well, I can't use the cv::CAP_INTEL_MFX for the api-reference parameter of the cv::VideoWriter. But there is also the cv::CAP_OPENCV_MJPEG, which works fine for the MJPG codec (not all video container are supported, I use .avi, sadly .mkv was not working with this configuration). If the user does not use MJPG as a codec I use cv::CAP_ANY, and opencv decides what is to use.

So,

cv::VideoWriter videoWriter(path, cv::CAP_OPENCV_MJPEG, fourcc, fps, *size);

works pretty well, even on my old system.

Unfortunately I never changed the api-reference parameter before, only from ffmpeg to gstreamer, I read in the doc of opencv only the last line "cv::CAP_FFMPEG or cv::CAP_GSTREAMER." and I did not see that there is an "e.g." before... Thank you @Micka to make me read again.

P.S. for my performance problem with cv::imshow I changed from

cv::namedWindow(WINDOW_NAME, cv::WINDOW_NORMAL);

to

cv::namedWindow(WINDOW_NAME, cv::WINDOW_OPENGL);

Which obviously uses OpenGL, and does a better job. Also changing from cv::Mat to cv::UMat can speed up the performance, see here

-------------- EDIT better solution ----------------

Since I still had problems with the OpenCV VideoWriter for some systems, I was looking for another solution. Now I write the frames with FFMPEG. For FFMPEG I can use the GPU or CPU depending on the codec I use. If FFMPEG is installed via snapd (Ubuntu 18.04) it comes with cuda enabled by default:

sudo snap install ffmpeg --devmode

(--devmode is optional, but I had problems writing files on specific location, this was the only way for me to fix it)

And here is my code:

//this string is automatically created in my program, depending on user input and the parameters of the input frames

string ffmpegCommand = "ffmpeg -y -f rawvideo -vcodec rawvideo -framerate 50 -pix_fmt bgr24 -s 1920x1080 -i - -c:v h264_nvenc -crf 14 -maxrate:v 10M -r 50 myVideoFile.mkv";

FILE *pipeout = popen(ffmpegCommand.data(), "w");

int id = metaDataWriter.insertNow(path);

//loop will be stopped from another thread
while (this->isRunning) {
    //this->frames is a stack with cv::Mat elements in the right order
    //it is filled by another thread
    while (!this->frames.empty()) {


        cv::Mat mat = frames.front();
        frames.pop();
        fwrite(mat.data, 1, s, pipeout);

    }  
}

fflush(pipeout);
pclose(pipeout);

So a file (pipeout) is used to write the mat.data to ffmpeg, ffmpeg itself is doing the encoding and file writing. To the parameters:

-y = Overwrite output files without asking

-f = format, in this case used for input rawvideo

-vcodec = codec for input which is rawvideo as well, because the used cv::Mat.data has no compression/codec

-framerate = the input framerate I receive from my grabber card/OpenCv

-pix_fmt = the format of my raw data, in this case bgr24, so 8 bit each channel, because I use a regular OpenCV bgr cv::Mat

-s = size of each frame, in my case 1920x1080

-i = input, in this case we read from the stdinput you can see it here "-", so the file (pipeout) is captured by ffmpeg

-c:v = output codec, so this is to encode the video, here h264_nvenc is used, which is a GPU codec

-r = frame output rate, also 50 in this case myVideoFile.mkv = this is just the name of the file which is produced by ffmpeg, you can change this file and path

Additional parameters for higher quality: -crf 14 -maxrate:v 10M

This works very good for me and uses my hardware acceleration of the GPU or with another codec in charge the CPU. I hope this helps other developers as well.

Why opencv videowriter is so slow?

1 Answers