How can I convert a stream of ArrayBuffer to a video?

Viewed 1829

I am generating a video stream from my webcam using ffmpeg from my Raspberry Pi, by using the following command:

ffmpeg -nostats -loglevel level+info -f v4l2 -video_size 640x480 -i /dev/video0 -f mpegts -codec:v mpeg1video -s 640x480 -b:v 500k -bf 0 http://192.168.10.6:3030/stream

My API server receives the stream data and send it through socket.io to my (React) client. I can receive the data on my client, but it's a flux of ArrayBuffer; I'm not sure how to transform this data to display an actual video on canvas. Any idea?

The API route that handles the stream:

import logger from '@src/startup/logging'
import { Router } from 'express'
import { io } from '@src/index'

const stream = Router()

stream.post('/', (req, res) => {
  logger.info('received')

  req.on('data', (chunk: string) => {
    logger.info(chunk.length)
    io.sockets.emit('data', chunk)
  })

  req.on('end', () => {
    logger.info('Stream Terminated')
    res.end()
  })

  // res.status(200).send('video')
})

export default stream

The client page that should transform the data in images (doesn't work):

import { useEffect, useState } from 'react'
import socketIoClient from './socketIoClient'

export default function App(): JSX.Element {
  const [srcBlob, setSrcBlob] = useState(null)

  useEffect(() => {
    const handleSocketIo = async () => {
      const socket = await socketIoClient()

      socket.on('greeting', (data: string) => {
        console.log(data)
      })

      socket.on('data', (data: ArrayBuffer) => {
        console.log(data)
        // I just copied this from another example, but it does not display anything
        const arrayBufferView = new Uint8Array(data)
        const blob = new Blob([arrayBufferView], { type: 'image/jpeg' })
        const urlCreator = window.URL || window.webkitURL
        setSrcBlob(urlCreator.createObjectURL(blob))
      })

      socket.emit('greeting', 'hello from client')
    }

    handleSocketIo()
  }, [])

  if (!srcBlob) {
    return <div>Loading...</div>
  }

  return <img src={srcBlob} />
}

This is part of the output I get when I use ffmpeg:

[info] Input #0, video4linux2,v4l2, from '/dev/video0':
[info]   Duration: N/A, start: 174842.704640, bitrate: 147456 kb/s
[info]     Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 640x480, 147456 kb/s, 30 fps, 30 tbr, 1000k tbn, 1000k tbc
[info] Stream mapping:
[info]   Stream #0:0 -> #0:0 (rawvideo (native) -> mpeg1video (native))
[info] Press [q] to stop, [?] for help
[info] Output #0, mpegts, to 'http://192.168.10.6:3030/stream':
[info]   Metadata:
[info]     encoder         : Lavf58.20.100
[info]     Stream #0:0: Video: mpeg1video, yuv420p, 640x480, q=2-31, 500 kb/s, 30 fps, 90k tbn, 30 tbc
[info]     Metadata:
[info]       encoder         : Lavc58.35.100 mpeg1video
[info]     Side data:
[info]       cpb: bitrate max/min/avg: 0/0/500000 buffer size: 0 vbv_delay: -1

This is an example of the data received by the client (console.log):

enter image description here

0 Answers
Related