How to merge video and camera recording together in browser (Chrome especially)?

user_3127828 0
Viewed 1022

Goal

I want to record/generate a video in browser (Chrome especially) with a custom video (e.g. .mp4, .webm) and camera recording side-by-side.

--------------------------------------------------
|                        |                       |
|  Some Custom Video     |       My Camera       |
|                        |                       |
--------------------------------------------------

What is working

I can use MediaRecorder to record my camera, and play the recording side-by-side with my video, and download the recorded video as a webm.

Challenge

I'm facing difficulty merging the video and camera recording into a single video file side-by-side.

My investigation

MultiStreamMixer

I first looked into MultiStreamMixer and built a demo with it (See codepen).

enter image description here

The issue with it is that it stretches the video content to fit them in the same size. I can specify different width/height for those two streams but it doesn't work as expected. My camera got cropped.

enter image description here

Custom Mixer

I took a look at the source code of MultiStreamMixer and found the issue was because of its simple layout logic. So I took its source code as a reference and build my custom mixer. See codepen.

The way it works:

  • We first render the streams one by one to an offscreen canvas.
  • Capture the stream from the canvas as the output video stream
  • Audio stream is generated separately using AudioContext, createMediaStreamSource, createMediaStreamDestination etc.
  • Merge the audio and video streams and output as a single stream.
  • Use MediaRecorder to record the mixed stream.

It adds black margins to video/camera and won't stretch the videos.

enter image description here

However, I found the recording is very blurry if you wave your hand in front of your camera while recording.

enter image description here

Initially I thought it was because I didn't set some setting correctly to the canvas. But later I found that even the my MultiStreamMixer demo or the WebRTC demo (You can't see the text on the teapot clearly in the recording) generates blurry video with canvas.

I'm asking in webrtc group to see if I can get around this issue. Meanwhile I looked into ffmpeg.js

ffmpeg.js

I think this would "work" but the file is too large. It's impratical to let the customer wait for this 23MB JS file to be downloaded.

Other ways that I haven't tried

The above are my investigations thus far.

Another idea is to play the video and recorded video side-by-side and use screen recording API to record the merged version. (Example). But this would require the customer to wait for the same amount of time as the initial recording to get the screen/tab recorded.

Uploading the video to server and doing the work in server would be my last resort.

mediarecorder ffmpeg video canvas javascript
2 Answers

The real answer to this is that to merge two video files you're going to have to take the two seperate files and run something over them to merge them. You can't take two video input streams (webcam and video, in this case) and just drop one next to the other in that way.

What you could do is:

  1. Keep the video's separate and just overlay one html5 player ontop of the other. Should work but synchronization would be a pain.
  2. Use ffmpeg.js (really not that bad if you cache it locally, compress it and make the loading look pretty)
  3. Send the video chunks to a server to be transcoded
user_19986920

There is an old expression that of "cheap, fast, and good", you can only pick 2 of the 3. If you increase the information content of your video (say, by waving your hand in front of the camera), one of the following must happen: the frame rate has to go down, the quality has to go down, the output bitrate has to go up.

Webrtc is designed to support video conferencing, a scenario in which high frame rates are more important than high quality. You may have to manipulate the SDP to request higher bit rates. It may also be the wrong tool for the job.

user_43272300
Related