FFMPEG's sws_scale for converting RGB to YUV420 image is extremely slow

user_14123002 0
Viewed 873

I am creating a basic application for recording screen activity using FFMPEG Library calls. My Program flow is as below - Fetch input data from framebuffer (in RGB format) --> Converting to YUV420 format and scaling to desirable resolution -> Encode the frame and Send to Muxer for MPEG2 convesrion.

Input data to my program is raw frambuffer data in RGB Format. I am using FFMPEG's sws_scale api for converting RGB to YUV420 image for encoding. Below is the code for converting the pixel format-

static int get_frame_buffer_data(AVFrame *pict, int frame_index, int width,
                      int height, enum AVPixelFormat pix_fmt, char *rawFrame)
 {
     struct SwsContext *sws_ctx = NULL;
     int ret = 0;
     rfbLog("[%s:%d]before conv_frame alloc:::pix_fmt = %d width = %d height = %d\n",__func__,__LINE__,pix_fmt,width,height);
     //picture->data[0] = (uint8_t*)&frameBuffer[0];
    picture->data[0] = (uint8_t*)&rawFrame[0];
     sws_ctx = sws_getCachedContext(sws_ctx,picture->width, picture->height, picture->format,width, height, pix_fmt,SWS_BICUBIC, NU       LL, NULL, NULL);
     if (!sws_ctx)
     {
         rfbLog("[%s:%d]Cannot initialize the conversion context\n",__func__,__LINE__);
         av_frame_free(&picture);
         sws_freeContext(sws_ctx);
         return -1;
     }
     rfbLog("[%s:%d]before sws_scale::: picture->linesize[0]=%d picture->height=%d pict->linesize = %d\n",__func__,__LINE__,picture       ->linesize[0],picture->height,pict->linesize[0]);
     ret = sws_scale(sws_ctx, (const uint8_t * const *)picture->data, picture->linesize, 0, picture->height, pict->data, pict->line       size);
     rfbLog("[%s:%d]after sws_scale::: picture->linesize[0]=%d picture->height=%d pict->linesize = %d returned height = %d\n",__fun       c__,__LINE__, picture->linesize[0],picture->height,pict->linesize[0], ret);
     if (ret < 0)
     {
         rfbLog("[%s:%d]could not convert to yuv420\n",__func__,__LINE__);
         sws_freeContext(sws_ctx);
         av_frame_free(&picture);
         return -1;
     }
     sws_freeContext(sws_ctx);
     return 0;
 }

I noticed that adding this code is making the application very slow. Profiling data gave me the information that sws_scale is taking very much time for converting the data to YUV420 . It is taking almost around 200 ms, which is making CPU time utilization very high and making my application unresponsive sometime.

Can we optimize this or use any alternative solution for conversion and how can we achieve that?

ffmpeg
2 Answers

"Optimization" in software means achieving the same result with fewer instructions. You call one function sws_scale So unless you what to modify the ffmpeg source code, The only "optimization" you can do is at compile time. What options were set what ffmpeg was compiled? Try recompiling with -O3.

Other options:

Switch to zimg, Its a little bit faster, but not much because color conversion is a complicated process.

User a faster computer. 200ms is pretty slow, Unless you are scaling very large images, I suspect the is stunning on an underpowered CPU.

Use more threads. Assuming the CPU has more than one core (Or the machine has more than one CPU) You can run multiple instance of sws_scale Each frame will still be 200ms, But if you can do more at the same time, it will bring the average down.

user_2107850

I don't think that the problem is on sws_scale and I don't believe that you can actually optimize it as far as the convert happens on CPU.

If you really want good performance you should look for Post-Process hardware (GPU) acceleration and pixel shaders (based on the OS you use, eg DirectX for Windows).

user_123442630
Related