I am using OpenTok to build a live video platform. It generates webm files from each users stream.
I am using FFMPEG to convert webm (WEBRTC) videos to MP4s to edit in my NLE. The problem I am having is my audio is drifting. I THINK it is because the user pauses the audio during the stream. This is the command i'm running
ffmpeg -acodec libopus -i 65520df3-1033-480e-adde-1856d18e2352.webm -max_muxing_queue_size 99999999 65520df3-1033-480e-adde-1856d18e2352.webm.new.mp4
The problem is I think, whenever the user muted themselves, there are no frames. But the PTS is in tact.
This is from the OpenTok documentation (my WebRTC platform)
Audio and video frames may not arrive with monotonic timestamps; frame rates are not always consistent. This is especially relevant if either the video or audio track is disabled for a time, using one of publishVideo or publishAudio publisher properties.
Frame presentation timestamps (PTS) are written based on NTP timestamps taken at the time of capture, offset by the timestamp of the first received frame. Even if a track is muted and later unmuted, the timestamp offset should remain consistent throughout the duration of the entire stream. When decoding in post-processing, a gap in PTS between consecutive frames will exist for the duration of the track mute: there are no "silent" frames in the container.
How can I convert these files and have them play in sync? Note, when I play in QuickTime or VLC, the files are synced correctly.
EDIT I've gotten pretty close with this command:
ffmpeg -acodec libopus -i $f -max_muxing_queue_size 99999999 -vsync 1 -af aresample=async=1 -r 30 $f.mp4
But every once in a while, I get a video where the audio starts right away, and they wont actually be talking in the video until half-way thought the video. My guess is they muted themselves during the video conference... so in some cases audio is 5-10 mins ahead. Again, plays fine in quicktime, but pulled into my NLE, its way off.