Real time audio streaming from ffmpeg to browser (am I missing something?)

Question

I have tried a couple of solutions already, but nothing works for me. I want to stream audio from my PC to another computer with almost zero latency. Things are working fine so far in a sense of lagging and everything, sound is clear and not choppy at all, but there is something like a delay between the moment when audio starts playing on my PC and remote PC. For example when I click on Youtube 'play' button audio starts playing only after 3-4 seconds on the remote machine. The same when I click 'Pause', the sound on the remote PC stops after a couple of seconds.

I've tried to use websockets\plain audio tag, but no luck so far.

For example this is my solution by using websockets and pipes:

def create_pipe():
    return win32pipe.CreateNamedPipe(r'\.\pipe\__audio_ffmpeg', win32pipe.PIPE_ACCESS_INBOUND,
                                     win32pipe.PIPE_TYPE_MESSAGE |
                                     win32pipe.PIPE_READMODE_MESSAGE |
                                     win32pipe.PIPE_WAIT, 1, 1024 * 8, 1024 * 8, 0, None)


async def echo(websocket):
    pipe = create_pipe()
    win32pipe.ConnectNamedPipe(pipe, None)
    while True:
        data = win32file.ReadFile(pipe, 1024 * 2)
        await websocket.send(data[1])


async def main():
    async with websockets.serve(echo, "0.0.0.0", 7777):
        await asyncio.Future()  # run forever


if __name__ == '__main__':
    asyncio.run(main())

The way I start ffmpeg

.\ffmpeg.exe -f dshow -i audio="Stereo Mix (Realtek High Definition Audio)" -acodec libmp3lame  -ab 320k -f mp3 -probesize 32 -muxdelay 0.01 -y \.\pipe\__audio_ffmpeg

On the JS side the code is a little bit long, but essentially I am just reading a web socket and appending to buffer

this.buffer = this.mediaSource.addSourceBuffer('audio/mpeg')

Also as you see I tried to use -probesize 32 -muxdelay 0.01 flags, but no luck as well

I tried to use plain tag as well, but still - this couple-of-seconds delay exists

What can I do? Am I missing something? Maybe I have to disable buffering somewhere?

IT goldman · Answer

I have some code, but all I learned was from https://webrtc.github.io/samples/ website and some from MDN. It's pretty simple.

The idea is to connect 2 peers using a negotiating server just for the initial connection. Afterwards they can share streams (audio, video, data). When I say peers I mean client computers like browsers.

So here's an example for connecting, and broadcasting and of course receiving.

Now for some of my code.

a sketch of the process

note: the same code is used for connecting to and connecting from. this is how my app works bcz it's kind of like a chat. ClientOutgoingMessages and ClientIncomingMessages are just my wrapper around sending messages to server (I use websockets, but it's possible also ajax).

Start: peer initiates RTCPeerConnection and sends an offer via server. also setup events for receiving. The other peer is notified of the offer by the server, then sends answer the same way (should he choose to) and finally the original peer accepts the answer and starts streaming. Among this there is another event about candidate I didn't even bothered to know what it is. It works without knowing it.


function create_pc(peer_id) {

    var pc = new RTCPeerConnection(configuration);
    var sender

    var localStream = MyStreamer.get_dummy_stream();
    for (var track of localStream.getTracks()) {
        sender = pc.addTrack(track, localStream);
    }

    // when a remote user adds stream to the peer connection, we display it
    pc.ontrack = function (e) {
        console.log("got a remote stream")
        remoteVideo.style.visibility = 'visible'
        remoteVideo.srcObject = e.streams[0]
    };

    // Setup ice handling
    pc.onicecandidate = function (ev) {
        if (ev.candidate) {
            ClientOutgoingMessages.candidate(peer_id, ev.candidate);
        }
    };

    // status
    pc.oniceconnectionstatechange = function (ev) {
        var state = pc.iceConnectionState;
        console.log("oniceconnectionstatechange: " + state)
    };

    MyRTC.set_pc(peer_id, {
        pc: pc,
        sender: sender
    });

    return pc;
}

function offer_someone(peer_id, peer_name) {


    var pc = MyRTC.create_pc(peer_id)
    pc.createOffer().then(function (offer) {
        ClientOutgoingMessages.offer(peer_id, offer);
        pc.setLocalDescription(offer);
    });

}

function answer_offer(peer_id) {
    var pc = MyRTC.create_pc(peer_id)
    var offer = MyOpponents.get_offer(peer_id)
    pc.setRemoteDescription(new RTCSessionDescription(offer));
    pc.createAnswer().then(function (answer) {
        pc.setLocalDescription(answer);
        ClientOutgoingMessages.answer(peer_id, answer);
        // alert ("rtc established!")
        MyStreamer.stream_current();
    });
}

handling messages from server


    offer: function offer(data) {
        if (MyRTC.get_pc(data.connectedUser)) {
            // alert("Not accepting offers already have a conn to " + data.connectedUser)
            // return;
        }
        MyOpponents.set_offer(data.connectedUser, data.offer)
        
    },

    answer: function answer(data) {
        var opc = MyRTC.get_pc(data.connectedUser)

        opc && opc.pc.setRemoteDescription(new RTCSessionDescription(data.answer)).catch(function (err) {
            console.error(err)
            // alert (err)
        });
        // alert ("rtc established!")

        MyStreamer.stream_current();
    },

    candidate: function candidate(data) {
        var opc = MyRTC.get_pc(data.connectedUser)
        opc && opc.pc.addIceCandidate(new RTCIceCandidate(data.candidate));
    },

    leave: function leave(data) {
        MyRTC.close_pc(data.connectedUser);
    },

Real time audio streaming from ffmpeg to browser (am I missing something?)

1 Answers