FFmpeg C audio video streams (mic, webcam) sync to mp4

Question

I am trying to capture audio and video using a microphone and a webcam into an mp4 file. The recorded file is playable but over the time the audio starts drifting away from video and for a longer period of time the gap increases. Both the audio and video is handled in separate threads, for the audio I am using audiofifo adapted from the transcode_acc.c example https://github.com/FFmpeg/FFmpeg/blob/master/doc/examples/transcode_aac.c
Here is how I am setting up the streams
Video:

video_output_codec_ctx = video_stream->codec;
video_output_codec_ctx->bit_rate = 2000000;
video_output_codec_ctx->codec_id = AV_CODEC_ID_MPEG4;
video_output_codec_ctx->width = 640;
video_output_codec_ctx->height = 480;
video_stream->time_base = (AVRational){1, fps};
video_output_codec_ctx->time_base = video_stream->time_base;
video_output_codec_ctx->pix_fmt = AV_PIX_FMT_YUV420P;
video_output_codec_ctx->codec_type = AVMEDIA_TYPE_VIDEO;

Audio:

audio_output_codec_ctx->channels       = OUTPUT_CHANNELS; // 2
audio_output_codec_ctx->channel_layout = av_get_default_channel_layout(OUTPUT_CHANNELS);
audio_output_codec_ctx->sample_rate    = audio_input_codec_ctx->sample_rate;
audio_output_codec_ctx->sample_fmt     = audio_output_codec->sample_fmts[0];
audio_output_codec_ctx->bit_rate       = OUTPUT_BIT_RATE; // 96000
audio_output_codec_ctx->strict_std_compliance = FF_COMPLIANCE_EXPERIMENTAL;

/* Set the sample rate for the container. */
audio_stream->time_base.den = audio_input_codec_ctx->sample_rate;
audio_stream->time_base.num = 1;

For the video pts, I increment the index by 1 once the frame is encoded and before sending the frame to the encoder I use rescale and also after receiving the frame,afterwards the packets are written via av_interleaved_write_frame().

output_frame->pts = av_rescale_q(video_frame_index, video_output_codec_ctx->time_base, video_input_format_ctx->streams[0]->time_base);

error = avcodec_send_frame(video_output_codec_ctx, output_frame);

error = avcodec_receive_packet(video_output_codec_ctx, &output_packet);

output_packet.stream_index = video_index;

output_packet.pts = av_rescale_q_rnd(output_packet.pts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));

output_packet.dts = av_rescale_q_rnd(output_packet.dts, video_input_format_ctx->streams[0]->time_base, output_format_context->streams[video_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));

output_packet.duration = ((output_format_context->streams[0]->time_base.den / output_format_context->streams[0]->time_base.num) / video_output_codec_ctx->time_base.den);

output_packet.pos = -1;  

video_frame_index++;

For the audio pts, I increment by frame->nb_samples once the frame is encoded I use rescale, afterwards the packets are written via av_interleaved_write_frame().

frame->pts = aud_pts;
aud_pts += frame->nb_samples;

error = avcodec_send_frame(audio_output_codec_ctx, frame);

error = avcodec_receive_packet(audio_output_codec_ctx, &output_packet);

output_packet.stream_index = audio_index;

output_packet.pts = av_rescale_q_rnd(output_packet.pts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));

output_packet.dts = av_rescale_q_rnd(output_packet.dts, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base, (AVRounding)(AV_ROUND_NEAR_INF|AV_ROUND_PASS_MINMAX));

output_packet.duration = av_rescale_q(output_packet.duration, audio_output_codec_ctx->time_base, output_format_context->streams[audio_index]->time_base);

I am new to the FFmpeg C API, have tried various resources / posts over the internet but still haven't been able to sync the audio and video in a robust way. Here are few question that I want to understand which will help me to resolve this. Any thoughts are really appreciated on this.

Can the FFmpeg C API handle sync internally or this is something which needs to be handled from the caller side?
Am I setting up the PTS correctly both for audio and video in the first place? I have noticed that when I use fps lower than 20, I get Invalid pts (66667) <= last (66667) Operation not permitted from the encoder. This must be something wrong with the way I am currently setting the video PTS. How can I set up the video PTS to handle lower fps?
I am also trying to adopt the idea of clocks sync from dranger's tutorial, not sure whether this would be suitable for my use case, things like where to setup the audio and video clocks as he used only decoders, for the audio I am using fifo and not sure how to adjust the samples based on the clocks sync, also the way to call and setup the refresh timer?
Is there a better mechanism for creating a robust sync for my use case which can handle both audio and video if they go out of sync, would be great to have an idea about the samples and frames adjustment based on that?

FFmpeg C audio video streams (mic, webcam) sync to mp4

0 Answers