cv2.VideoWriter issues

user_19657208 0
Viewed 106

I'm looking to record a Twitch Livestream by feeding it the direct livestream url using streamlink.streams(url) (which returns a .m3u8 url). With this, I have no problem reading the stream and even writing a few images from it, but when it comes to writing it as a video, I get errors.

P.S.: Yes, I know there's other options like Streamlink and yt-dwl, but I want to operate solely in python, not using CLI... which I believe those two are only dealing with (for recording).

Here's what I currently have:

if streamlink.streams(url):
    stream = streamlink.streams(url)['best']
    stream = str(stream).split(', ')
    stream = stream[1].strip("'")
    cap = cv2.VideoCapture(stream)
    gst_out = "appsrc ! video/x-raw, format=BGR ! queue ! nvvidconv ! omxh264enc ! h264parse ! qtmux ! filesink location=stream "
    out = cv2.VideoWriter(gst_out, cv2.VideoWriter_fourcc(*'mp4v'), 30, (1920, 1080))
    while True:
        _, frame = cap.read()
        out.write(frame)

For this code, I get this error msg:

[tls @ 0x1278a74f0] Error in the pull function.

And if I remove gst_out and feed stream instead as well as moving cap and out into the while loop like so:

if streamlink.streams(url):
    stream = streamlink.streams(url)['best']
    stream = str(stream).split(', ')
    stream = stream[1].strip("'")
    while True:
        cap = cv2.VideoCapture(stream)
        _, frame = cap.read()
        out = cv2.VideoWriter(stream, cv2.VideoWriter_fourcc(*'mp4v'), 30, (1920, 1080))
        out.write(frame)

I get:

OpenCV: FFMPEG: tag 0x7634706d/'mp4v' is not supported with codec id 12 and format 'hls / Apple HTTP Live Streaming'

What am I missing here?

http-live-streaming video-processing ffmpeg opencv python
1 Answers

The fist part uses GStreamer syntax, and OpenCV for Python is most likely not built with GStreamer.
The answer is going to be focused on the second part (also because I don't know GStreamer so well).

There are several issues:

  • cap = cv2.VideoCapture(stream) should be before the while True loop.
  • out = cv2.VideoWriter(stream, cv2.VideoWriter_fourcc(*'mp4v'), 30, (1920, 1080)) should be before the while True loop.
  • The first argument of cv2.VideoWriter should be MP4 file name, and not stream.
  • For getting a valid output file, we have to execute out.release() after the loop, but the loop may never end.

  • It is recommended to get frame size and rate of the input video, and set VideoWriter accordingly:

     width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
 height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
 fps = int(cap.get(cv2.CAP_PROP_FPS))

 video_file_name = 'output.mp4'

 out = cv2.VideoWriter(video_file_name, cv2.VideoWriter_fourcc(*'mp4v'), fps, (width, height))  # Open video file for writing

  • It is recommended to break the loop if ret is False:

     ret, frame = cap.read()

 if not ret:
     break

  • One option to end the recording is when user press Esc key.
    Break the loop if cv2.waitKey(1) == 27.
    cv2.waitKey(1) is going to work only after executing cv2.imshow.
    A simple solution is executing cv2.imshow every 30 frames (for example).

     if (frame_counter % 30 == 0):
     cv2.imshow('frame', frame)  # Show frame every 30 frames (for testing)

 if cv2.waitKey(1) == 27:  # Press Esc for stop recording (cv2.waitKey is going to work only when cv2.imshow is used).
     break

Complete code sample:

from streamlink import Streamlink
import cv2

def stream_to_url(url, quality='best'):
    session = Streamlink()
    streams = session.streams(url)

    if streams:
        return streams[quality].to_url()
    else:
        raise ValueError('Could not locate your stream.')


url = 'https://www.twitch.tv/noraexplorer'  # Need to login to twitch.tv first (using the browser)...
quality='best'

stream_url = stream_to_url(url, quality)  # Get the video URL
cap = cv2.VideoCapture(stream_url, cv2.CAP_FFMPEG)  # Open video stream for capturing

# Get frame size and rate of the input video
width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
fps = int(cap.get(cv2.CAP_PROP_FPS))


video_file_name = 'output.mp4'

out = cv2.VideoWriter(video_file_name, cv2.VideoWriter_fourcc(*'mp4v'), fps, (width, height))  # Open video file for writing


frame_counter = 0
while True:
    ret, frame = cap.read()
    
    if not ret:
        break

    if (frame_counter % 30 == 0):
        cv2.imshow('frame', frame)  # Show frame every 30 frames (for testing)

    out.write(frame)  # Write frame to output.mp4

    if cv2.waitKey(1) == 27:  # Press Esc for stop recording (cv2.waitKey is going to work only when cv2.imshow is used).
        break

    frame_counter += 1

cap.release()
out.release()
cv2.destroyAllWindows()

Testing the setup using FFplay and subprocess module:

from streamlink import Streamlink
import subprocess

def stream_to_url(url, quality='best'):
    session = Streamlink()
    streams = session.streams(url)

    if streams:
        return streams[quality].to_url()
    else:
        raise ValueError('Could not locate your stream.')


#url = 'https://www.twitch.tv/noraexplorer'  # Need to login to twitch.tv first (using the browser)...
url = 'https://www.twitch.tv/valorant'
quality='best'

stream_url = stream_to_url(url, quality)  # Get the video URL

subprocess.run(['ffplay', stream_url])

Update:

Using ffmpeg-python for reading the video, and OpenCV for recording the video:

In cases where cv2.VideoCapture is not working, we may use FFmpeg CLI as sub-process.
ffmpeg-python module is Python binding for FFmpeg CLI.
Using ffmpeg-python is almost like using subprocess module, it used here mainly for simplifying the usage of FFprobe.

Using FFprobe for getting video frames resolution and framerate (without using OpenCV):

p = ffmpeg.probe(stream_url, select_streams='v');
width = p['streams'][0]['width']
height = p['streams'][0]['height']
r_frame_rate = p['streams'][0]['r_frame_rate']  # May return 60000/1001

if '/' in r_frame_rate:
    fps = float(r_frame_rate.split("/")[0]) / float(r_frame_rate.split("/")[1])  # Convert from 60000/1001 to 59.94
elif r_frame_rate != '0':
    fps = float(r_frame_rate)
else:
    fps = 30  # Used as default

Getting the framerate may be a bit of a challenge...

Note: ffprobe CLI should be in the execution path.

Start FFmpeg sub-process with stdout as pipe:

ffmpeg_process = (
    ffmpeg
    .input(stream_url)
    .video
    .output('pipe:', format='rawvideo', pix_fmt='bgr24')
    .run_async(pipe_stdout=True)
)

Note: ffmpeg CLI should be in the execution path.

Reading a frame from the pipe, and convert it from bytes to NumPy array:

in_bytes = ffmpeg_process.stdout.read(width*height*3)
frame = np.frombuffer(in_bytes, np.uint8).reshape([height, width, 3])

Closing FFmpeg sub-process:
Closing stdout pipe ends FFmpeg (with "broken pipe" error).

ffmpeg_process.stdout.close()
ffmpeg_process.wait()  # Wait for the sub-process to finish

Complete code sample:

from streamlink import Streamlink
import cv2
import numpy as np
import ffmpeg

def stream_to_url(url, quality='best'):
    session = Streamlink()
    streams = session.streams(url)

    if streams:
        return streams[quality].to_url()
    else:
        raise ValueError('Could not locate your stream.')


#url = 'https://www.twitch.tv/noraexplorer'  # Need to login to twitch.tv first (using the browser)...
url = 'https://www.twitch.tv/valorant'
quality='best'

stream_url = stream_to_url(url, quality)  # Get the video URL

#subprocess.run(['ffplay', stream_url])  # Use FFplay for testing

# Use FFprobe to get video frames resolution and framerate.
################################################################################
p = ffmpeg.probe(stream_url, select_streams='v');
width = p['streams'][0]['width']
height = p['streams'][0]['height']
r_frame_rate = p['streams'][0]['r_frame_rate']  # May return 60000/1001

if '/' in r_frame_rate:
    fps = float(r_frame_rate.split("/")[0]) / float(r_frame_rate.split("/")[1])  # Convert from 60000/1001 to 59.94
elif r_frame_rate != '0':
    fps = float(r_frame_rate)
else:
    fps = 30  # Used as default

#cap = cv2.VideoCapture(stream_url, cv2.CAP_FFMPEG)  # Open video stream for capturing
#width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
#height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
#fps = int(cap.get(cv2.CAP_PROP_FPS))
################################################################################


# Use FFmpeg sub-process instead of using cv2.VideoCapture
################################################################################
ffmpeg_process = (
    ffmpeg
    .input(stream_url, an=None)  # an=None applies -an argument (used for ignoring the input audio - it is not required, just more elegant).
    .video
    .output('pipe:', format='rawvideo', pix_fmt='bgr24')
    .run_async(pipe_stdout=True)
)
################################################################################


video_file_name = 'output.mp4'

out = cv2.VideoWriter(video_file_name, cv2.VideoWriter_fourcc(*'mp4v'), fps, (width, height))  # Open video file for writing


frame_counter = 0
while True:
    #ret, frame = cap.read()    
    in_bytes = ffmpeg_process.stdout.read(width*height*3)  # Read raw video frame from stdout as bytes array.
    
    if len(in_bytes) < width*height*3:  #if not ret:
        break

    frame = np.frombuffer(in_bytes, np.uint8).reshape([height, width, 3])  # Convert bytes array to NumPy array.

    if (frame_counter % 30 == 0):
        cv2.imshow('frame', frame)  # Show frame every 30 frames (for testing)

    out.write(frame)  # Write frame to output.mp4

    if cv2.waitKey(1) == 27:  # Press Esc for stop recording (cv2.waitKey is going to work only when cv2.imshow is used).
        break

    frame_counter += 1

#cap.release()
ffmpeg_process.stdout.close()  # Close stdout pipe (it also closes FFmpeg).
out.release()
cv2.destroyAllWindows()
ffmpeg_process.wait()  # Wait for the sub-process to finish

Note:
In case you care about the quality of the recorded video, using cv2.VideoWriter is not the best choice...

user_49267570
Related