Why does OpenCV read video faster than FFMPEG?

tadejsv 0
Viewed 85

I noticed that OpenCV reads video frames almost 2x faster than FFMPEG.

Why is that? I thought all OpenCV does is call FFMPEG under the hood, possibly adding its own overhead.

Here's the code I use to obtain these results

import cv2
import time
import numpy as np

cap = cv2.VideoCapture("BigBuckBunny.mp4", apiPreference=cv2.CAP_FFMPEG)
frames = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

start = time.perf_counter()
while True:
    ret, frame = cap.read()
    if ret is False:
        break
    assert frame.shape == (720, 1280, 3)
    assert frame.dtype == np.uint8
end = time.perf_counter()

print(f"{frames/(end-start):.1f} frames per second")
# Output: 692.3 frames per second

cap.release()

For FFMPEG (using the python-ffmpeg library:

import ffmpeg
import numpy as np
import time

vid_info = ffmpeg.probe("BigBuckBunny.mp4")['streams'][1]
frames = int(vid_info['nb_frames'])

process1 = (
    ffmpeg
    .input("BigBuckBunny.mp4")
    .output('pipe:', format='rawvideo', pix_fmt='bgr24')
)
print(process1.compile())
# Output: ['ffmpeg', '-i', 'BigBuckBunny.mp4', '-f', 'rawvideo', '-pix_fmt', 'bgr24', 'pipe:']


process1 = process1.run_async(pipe_stdout=True)

start = time.perf_counter()
while True:
    in_bytes = process1.stdout.read(1280 * 720 * 3)
    if not in_bytes:
        break
    frame = np.frombuffer(in_bytes, np.uint8).reshape([720, 1280, 3])
end = time.perf_counter()
print(f"{frames/(end-start):.1f} frames per second")
# Output: 373.6 frames per second
process1.wait()

Here's information about the video (~10 minutes length)

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'BigBuckBunny.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 0
    compatible_brands: isomavc1mp42
    creation_time   : 2010-01-10T08:29:06.000000Z
  Duration: 00:09:56.47, start: 0.000000, bitrate: 2119 kb/s
  Stream #0:0(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 125 kb/s (default)
    Metadata:
      creation_time   : 2010-01-10T08:29:06.000000Z
      handler_name    : (C) 2007 Google Inc. v08.13.2007.
      vendor_id       : [0][0][0][0]
  Stream #0:1(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 1991 kb/s, 24 fps, 24 tbr, 24k tbn, 48 tbc (default)
    Metadata:
      creation_time   : 2010-01-10T08:29:06.000000Z
      handler_name    : (C) 2007 Google Inc. v08.13.2007.
      vendor_id       : [0][0][0][0]

And FFMPEG and OpenCV versions:

ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers

opencv-python-headless        4.6.0.66
ffmpeg opencv python
2 Answers

It may be due to the fact how the libraries are implemented and the overhead of pipes.

  1. cv2

cv2 implemented in c++ and uses libavcodec library (part of ffmpeg) for decoding frames:

https://github.com/opencv/opencv/blob/97c6ec6d49cb78321eafe6fa220ff80ebdc5e2f4/modules/videoio/src/cap_ffmpeg_impl.hpp#L1299

cv2 does not use ffmpeg as binary executable and uses the part of ffmpeg specifically designed for reading/decoding frames.

  1. ffmpeg-python is a wrapper around ffmpeg executable (which in turn uses libavcodec under the hood too) that calls ffmpeg with subprocess and returns results through the pipes.
u1234x12340

Good stuff. Interestingly, under Windows 10 & FFmpeg 5.0.1, my results are the reverse of yours (but much closer):

  • OpenCV: 491.9 frames per second
  • FFmpeg-Python: 519.4 frames per second

And out of curiosity, I tried my ffmpegio package to see how much overhead it incurs, it came out: 507.5 frames per second (code below) Both FFmpeg-Python and FFmpegIO scoring in the same ballpark indicates these numbers aren't fluke.

So, it's likely Win/Linux or 4.4/5.0 differences or both. I retested FFmpeg-Python with FFmpeg 4.4.1 and got: 428.1 frames per second (log reporting speed=17.9x down from x21.7. I'd say FFmpeg devs optimized the color conversion codebase in v5 releases to match or surpass OpenCV. (Or the tested OpenCV version uses the libav libraries associated with FFmpeg v5)

test_ffmpegio.py

import ffmpegio
import time

frames = ffmpegio.probe.video_streams_basic("BigBuckBunny.mp4")[0]['nb_frames']

print(frames)

with ffmpegio.open("BigBuckBunny.mp4",'rv') as f:
    start = time.perf_counter()
    for frame in f:
        pass # frame is 1x1280x720x3 rgb np array
    end = time.perf_counter()

print(f"{frames/(end-start):.1f} frames per second")
kesh0
Related