Javascript + Node - Possible to edit audio from mediastream on the server and send it back to the client?

Viewed 208

So I'm trying to manipulate the audio which comes from the client on the server and then return the newly manipulated audio back to the client. Of course this runs into the error that Node does not support the web-audio-api so I'm using the Node version of web-audio-api along with a WebRTC library.

As I'm new to the WebRTC world I've been building on from these examples which use the WebRTC library. Using the audio-video-loopback example as the starting point I've utilised some of the libraries none standard APIs to create an audio sink that allows me to directly access the samples from the client. For now I just want to change the volume so I'm just changing the values and pushing them into a new track which is how the doc (scroll down to Programmatic Audio) says to do this. At the end I just want to return to the newly created track which is done using the .replaceTrack method (which I believe retriggers a renegotiation).

Here's what I got so far for the server code (client is the same as the original example found in the link above):

const { RTCAudioSink, RTCAudioSource } = require("wrtc").nonstandard;

function beforeOffer(peerConnection) {
  const audioTransceiver = peerConnection.addTransceiver("audio");
  const videoTransceiver = peerConnection.addTransceiver("video");

  let { track } = audioTransceiver.receiver;

  const source = new RTCAudioSource();
  const newTrack = source.createTrack();
  const sink = new RTCAudioSink(track);

  const sampleRate = 48000;
  const samples = new Int16Array(sampleRate / 100); // 10 ms of 16-bit mono audio

  const dataObj = {
    samples,
    sampleRate,
  };

  const interval = setInterval(() => {
    // Update audioData in some way before sending.
    source.onData(dataObj);
  });

  sink.ondata = (data) => {
    // Do something with the received audio samples.
    const newArr = data.samples.map((el) => el * 0.5);
    dataObj[samples] = newArr;
  };

  return Promise.all([
    audioTransceiver.sender.replaceTrack(newTrack),
    videoTransceiver.sender.replaceTrack(videoTransceiver.receiver.track),
  ]);
}

Not surprisingly this doesn't work, I just get silence back even though the dataObj contains the correctly manipulated samples which is then passed to the newTrack when source.onData is called.

Is what I'm trying to do even possible server side? Any suggestions are welcome, like I said I'm very green with WebRTC.

0 Answers
Related