webrtc h264 video garbled chrome

Question

Hi I am trying to set up a live stream of a camera to stream to the chrome browser. My stream from the camera is encoded in h264 so I would like to directly send this to the browser using gstreamers webrtcbin element. I can get it to push video to the screen but the issue is that the video comes in garbled. I have tested it with the videotestsrc to see if directly encoding it and sending it to the browser would shed some light.

Here is what I see:

This is the pipeline:

PIPELINE_DESC = '''
webrtcbin name=sendrecv bundle-policy=max-bundle
 videotestsrc is-live=true pattern=snow ! videoconvert ! queue ! x264enc ! rtph264pay !
 queue ! application/x-rtp,media=video,encoding-name=H264,payload=96 ! sendrecv.
 audiotestsrc is-live=true wave=red-noise ! audioconvert ! audioresample ! queue ! opusenc ! rtpopuspay !
 queue ! application/x-rtp,media=audio,encoding-name=OPUS,payload=96 ! sendrecv.
'''

Is there something I am missing?

EDIT: Interestingly when trying a different pattern such as ball the output looks and behaves fine:

EDIT2: The video plays fine in firefox... hrmm... Does chrome have an issue displaying h264?

Can it have something to do with the keyframes?

EDIT3: I can see it perfectly when I run my pipeline locally. Could it be that I am dropping packets?

EDIT4: It works fine when I am running the same code on my laptop. But on my Pi I get a garbled mess. What can it be?

The cpu is at 70-100% (pi has 4 cores so sometimes I get more than 100%)
A networking issue?
keyframes are getting lost?
something else?

I guess what I am really asking is how would I send an h264 stream via webrtc to a browser using gstreamer?

Edit:Here is my solution:

from the machine where the video originates from:

gst-launch-1.0 -v videotestsrc ! x264enc speed-preset=ultrafast ! "video/x-h264,profile=constrained-baseline,width=1280,height=720,stream-format=byte-stream,level=(string)3.1" ! rtph264pay config-interval=1 ! udpsink port=7001

We can see that the level is set to 3.1 and we are encoding, but this will also work for video that is already in h264 such as directly from a camera.

the client/webbrowser would need this pipeline:

PIPELINE_DESC = '''
    webrtcbin name=sendrecv bundle-policy=max-bundle
        udpsrc port=7001 caps = "application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, payload=(int)96" ! rtph264depay ! 
        h264parse ! rtph264pay config-interval=-1 !
        queue ! application/x-rtp,media=video,encoding-name=H264,payload=96 ! rtpjitterbuffer ! sendrecv.'''

I am not entirely sure why it had this issue, I suspect it had something to do with the profile level, but the folks at the gstreamer irc mentioned that it would automatically negotiate the correct caps.

user_3734222 · Answer

You are sending white noise as a video pattern=snow this is impossible for the encoder to compress. White noise has very little redundancy in the signal so the encoder can not deal with it that good. (especially in a live streaming scenario).

In case of a ball pattern it works for you since the most of the video is just black (means a lot of redundancy in the signal) and its easy for the encoder to compress that one.

I guess your connection (data channel) is not good enough and therefore you see such a bad video. And from your screenshot you can clearly see that on the top left the video looks like it should and then it degrades in quality in the raster scan order.

To summarise:

can be a packet loss / networking issues
encoder does better job on your laptop (according to your Edit 4), and therefore less data is sent in that case.
you can try to tweak encoder parameters (not just using the default ones) to get it to work
sending white noise is usually not something you would like to do. Test it on your real content first. Try to find a worst case real content with a lot of movement too see if the problem still appears.

webrtc h264 video garbled chrome

1 Answers