I playing around websocket, media API, browsers and nodejs. My goal will be to enable two member to talk to each other in live.
The desired logic'
client-1_mic -> client-1_browser --> nodejs server -> client-2_browser -> client-2_speaker
I know, there is a way with WebRTC, but I would like to achieve via websockets.
Note: I use the following stuffs: Firefox v56/57 browser, NodeJS (v8), NPM, Ubuntu server.
My current solution right now able to communicate via websockets (channels also present), so a basic chat.
My serverside (server.js) part for communication:
socket.on('audio', function (data) {
if (socket.username in userTargets && userTargets[socket.username] in usersInRoom) {
io.sockets.connected[usersInRoom[userTargets[socket.username]].socket].emit('updatechat', socket.username, 'New audio arrived');
// send to itself
io.sockets.connected[socket.id].emit('updatechat', socket.username, 'To ' + userTargets[socket.username] + '> here is some new audio');
// Audio passing
io.sockets.connected[usersInRoom[userTargets[socket.username]].socket].emit('audio', socket.username, data);
} else {
io.sockets.in(socket.room).emit('updatechat', socket.username, 'Some audio arrived');
// Audio global spam @todo later add restrict agains the sender
io.sockets.in(socket.room).emit('audio', socket.username, data);
}
});
My client part(s), audio gathering (client.js)
media = mediaOptions.audio;
navigator.mediaDevices.getUserMedia(media.gUM).then(_stream => {
stream = _stream;
recorder = new MediaRecorder(stream);
chunks = [];
recorder.ondataavailable = e => {
chunks.push(e.data);
socket.emit('audio', {audioBlob: e.data}); // Sending only the audio blob
};
log('got media successfully');
}).catch(log);
And the audio retrieving part (client.js also)
socket.on('audio', function (senderUser, data) {
// Lets try to play the audio
try {
//audioBlob: ArrayBuffer { byteLength: 13542 }
//audioBlob: <Buffer 4f 67 67 53 00 00 80 3f 04 00 00 00 00 00 b8 51 43 3c 0a 00 00 00 e3 d9 48 13 20 80 81 80 81 80 81 80 81 80 81 7e 7f 78 78 7c 82 89 86 8a 81 80 81 80 ...
audioBufferContainer.push(data.audioBlob);
console.log('Blob chunk added!');
/*
// missing time duration & empty space filler for audio
// missing blob to arraybuffer conversion
audioCtx.decodeAudioData(data.audioBlob, function(myAudioBuffer) {
audioBufferSource.buffer = myAudioBuffer;
audioBufferSource.connect(audioCtx.audioDestination);
console.log('Audio buffer passed to the buffer source');
audioBufferSource.play();
});
*/
}
catch(e) {
console.error('Some error during replaying: ' + e.message);
}
});
If I just simply gathering the sent audio blobs (from beginin' - til' end) and just concat it, then i got the playable audio file (e.g.: the transfer is correct, there isn't data loss or corruption).
My questions:
- How I should handle the gathered/received audio chunks (blob)? Should I try to convert into audio array buffer then passing it to the audioContext and set the time/ ffset and play it (and maybe near it fill the gaps with empty sound)?
- Or there is any other lib/API what I can use for playing back a "stream" like this?
- Or should I just try to add to the client a <audio src="ws://<host>:<port>"> type of tag and the browser will handle then the replay?
- Or better to add some timer to the sender side and always send full, like half second or second long audio file what is complete (i mean has the header and metadata nd the closing as well) then just repeatedly spamming like different tracks and on client side just stacking into a source queue and playing like an album?
Note: I know there was a lib, called binaryJS, but it's not maintained (~5y) and the example is just not working
Anyone could give me some advice or hint what I don't get or where I made the mistake?
Ps.: I don't have extensive JS background, just other languages, so i'm not familiar with the async-type of structures or advanced node.js solutions either.