How to use ffmpeg in JavaScript to decode H.264 frames into RGB frames

user_1242028 0
Viewed 3468

I'm trying to compile ffmpeg into javascript so that I can decode H.264 video streams using node. The streams are H.264 frames packed into RTP NALUs so any solution has to be able to accept H.264 frames rather than a whole file name. These frames can't be in a container like MP4 or AVI because then the demuxer needs to needs the timestamp of every frame before demuxing can occur, but I'm dealing with a real time stream, no containers.

Streaming H.264 over RTP

Below is the basic code I'm using to listen on a udp socket. Inside the 'message' callback the data packet is an RTP datagram. The data portion of the data gram is an H.264 frame (P-frames and I-frames).

var PORT = 33333;
var HOST = '127.0.0.1';

var dgram = require('dgram');
var server = dgram.createSocket('udp4');

server.on('listening', function () {
    var address = server.address();
    console.log('UDP Server listening on ' + address.address + ":" + address.port);
});

server.on('message', function (message, remote) {
    console.log(remote.address + ':' + remote.port +' - ' + message);
    frame = parse_rtp(message);

    rgb_frame = some_library.decode_h264(frame); // This is what I need.

});

server.bind(PORT, HOST);

I found the Broadway.js library, but I couldn't get it working and it doesn't handle P-frames which I need. I also found ffmpeg.js, but could get that to work and it needs a whole file not a stream. Likewise, fluent-ffmpeg doesn't appear to support file streams; all of the examples show a filename being passed to the constructor. So I decided to write my own API.

My current solution attempt

I have been able to compile ffmpeg into one big js file, but I can't use it like that. I want to write an API around ffmpeg and then expose those functions to JS. So it seems to me like I need to do the following:

  1. Compile ffmpeg components (avcodec, avutil, etc.) into llvm bitcode.
  2. Write a C wrapper that exposes the decoding functionality and uses EMSCRIPTEN_KEEPALIVE.
  3. Use emcc to compile the wrapper and link it to the bitcode created in step 1.

I found WASM+ffmpeg, but it's in Chinese and some of the steps aren't clear. In particular there is this step:

emcc web.c process.c ../lib/libavformat.bc ../lib/libavcodec.bc ../lib/libswscale.bc ../lib/libswresample.bc ../lib/libavutil.bc \

:( Where I think I'm stuck

I don't understand how all the ffmpeg components get compiled into separate *.bc files. I followed the emmake commands in that article and I end up with one big .bc file.

2 questions

1. Does anyone know the steps to compile ffmpeg using emscripten so that I can expose some API to javascript?
2. Is there a better way (with decent documentation/examples) to decode h264 video streams using node?

emscripten fluent-ffmpeg node.js h.264 ffmpeg
2 Answers

To question 1: Just follow the official doc

Consider the case where you normally build with the following commands:

./configure
make

To build with Emscripten, you would instead use the following commands:

# Run emconfigure with the normal configure command as an argument.
./emconfigure ./configure

# Run emmake with the normal make to generate linked LLVM bitcode.
./emmake make

# Compile the linked bitcode generated by make (project.bc) to JavaScript.
# 'project.bc' should be replaced with the make output for your project (e.g. 'yourproject.so')
#  [-Ox] represents build optimisations (discussed in the next section).
./emcc [-Ox] project.bc -o project.js

To question 2: c/c++ libs can be called in a node environment. You can write some c/c++ glue code or use a proxy node module like node-ffi.

Using node-ffi to call exist libs may be easier. May it help :)

user_40069130

Easiest way, specially if you need to run it in web browsers, is to utilize Media Source Extension. I managed to do it in just 3 days. Moreover, it is used automatically GPU hardware ( Cuda, Intel Qsv, ... ) acceleration as far as browser builtin support. It is important if it runs in the real World app. I have tested yesterday, just 5% of my old i7 machine CPU to decode 4K ( 4 times more than 1080p ) IP Camera H.264 nal raw live streaming. I am not sure it under server side js like nodejs, but I expect the result is similar. Contact me if you need further more. About H.265 / HEVC, you need emscripten it partially of ffmpeg or x265 or OpenH265 similarily, with minimal bc size ( less than 1,2,3 M depending on config. ). Good luck...

user_56718530
Related