Fastest canvas to ffmpeg method

user_3362074 0
Viewed 1351

I have an Electron JS app where I draw to a canvas and then send each frame to ffmpeg to create a video. It works well, but performance is not great. This is how I currently do it:

  • I have an image stream (new PassThrough) that is piped to my ffmpeg process
  • When a frame is ready, I convert it to a blob with canvas.toBlob()
  • I convert the blob to arrayBuffer
  • And get a buffer with Buffer.from()
  • I write the buffer to the image stream with .write()

I have measured the time each step takes and the bottleneck is, by far, canvas.toBlob

Is there a way to do this entire process or the toBlob step faster? I have looked into HTMLCanvasElement.captureStream() but I don't think I can pipe that to ffmpeg.

I'm using P5js for drawing. It does not seem like there is a way to draw (more) directly to a blob or buffer. Even p5.Graphics seems to draw to a hidden canvas https://p5js.org/reference/#/p5.Graphics

Thanks

p5.js html5-canvas ffmpeg html javascript
1 Answers

You should be able to skip the blob conversion entirely. The raw image data from an HTML5 canvas is already available as a Uint8ClampedArray in RGBA format. canvas.toBlob does image encoding (by default to PNG) which takes a bunch of time, and then you waste extra time by converting that back to an arraybuffer. However, FFmpeg can be configured to take raw video in an uncompressed form, so the encoding and the roundtripping through blob and back to arraybuffer format is completely unnecessary (you'd need something like -vcodec rawvideo -pix_fmt rgba -s <width>x<height> in the command line options).

Get the right command line options for FFmpeg, and you should be able to use CanvasRenderingContext2D.getImageData().data and stream that directly to FFmpeg once per frame.

user_8814810
Related