Sound playback using FFmpeg and libsoundio in c++

Question

I am trying to make a video player desktop application in c++ using primarily FFmpeg and Qt6. As of for now, I can decode and play video frames correctly at the right speed, that is not a problem. I am now trying to get to playback audio, which is much harder than I expected it to be. I am using libsoundio for my audio library but the documentation is really poor and there are not many examples/tutorials on it. I am also a beginner when it comes to audio programming, although I understand the basics. First off, if anyone can recommend an audio library for this type of job let me know, but I would like to use open source libraries. Anyways, here is how I decode my audio data with FFmpeg. I'm not sure if I am doing it correctly as I could barely find documentation on that as well... I have a struct that contains all the information which is initiated through a function:

struct VideoReader
{
    bool valid;
    int width, height;
    int video_stream_index;
    int audio_stream_index;
    AVRational time_base;

    AVFormatContext* av_format_ctx;
    AVCodecContext* av_vi_codec_ctx;
    AVCodecContext* av_au_codec_ctx;
    AVPacket* packet;
    AVFrame* frame;
    SwsContext* sws_ctx;
    SwrContext* swr_ctx;
};

The function that initiates it is quite long and is not necessary to share but it populates all those values except for the sws_ctx and the swr_ctx.

Here is how I decode packets, this function is simplified, I left the video decoding out of it, ill take care of syncing once I can properly playback audio:

bool video_reader_read_au_frame(VideoReader *video_reader, unsigned char **frame_buffer)
{
    // Unpack video_reader
    auto& av_format_ctx         = video_reader->av_format_ctx;
    auto& av_codec_ctx          = video_reader->av_au_codec_ctx;
    auto& av_packet             = video_reader->packet;
    auto& av_frame              = video_reader->frame;
    auto& swr_ctx               = video_reader->swr_ctx;
    int&  audio_stream_index    = video_reader->audio_stream_index;

    // Decode the video frame data
    int response;
    while (av_read_frame(av_format_ctx, av_packet) >= 0)
    {
        last_frame = false;
        if (av_packet->stream_index != audio_stream_index)
        {
            av_packet_unref(av_packet);
            continue;
        }

        response = avcodec_send_packet(av_codec_ctx, av_packet);
        if (response < 0)
        {
            Logger::error("Could not decode packet.");
            return false;
        }

        response = avcodec_receive_frame(av_codec_ctx, av_frame);
        if (response == AVERROR(EAGAIN) || response == AVERROR_EOF)
        {
            av_packet_unref(av_packet);
            continue;
        }
        else if (response < 0)
        {
            Logger::error("Could not decode packet.");
            return false;
        }
        av_packet_unref(av_packet);
        break;
    }

    // Initialize SwrContext
    if (!swr_ctx) {
        swr_ctx = swr_alloc_set_opts(nullptr,
                                     av_codec_ctx->channel_layout, AV_SAMPLE_FMT_FLT,
                                     av_codec_ctx->sample_rate, av_codec_ctx->channel_layout,
                                     av_codec_ctx->sample_fmt,  av_codec_ctx->sample_rate,
                                     0, nullptr);
        if (!swr_ctx)
        {
            Logger::error("Could not create SwrContext.");
            return false;
        }

        if (swr_init(swr_ctx) < 0)
        {
            Logger::error("Could not initialize SwrContext.");
            return false;
        }
    }


    const int MAX_BUFFER_SIZE = av_samples_get_buffer_size(nullptr, av_frame->channels, av_frame->nb_samples, AV_SAMPLE_FMT_FLT, 1);
    *frame_buffer = (unsigned char*)av_malloc(MAX_BUFFER_SIZE);
    swr_convert(swr_ctx, frame_buffer, av_frame->nb_samples,
                (const unsigned char**)av_frame->data, av_frame->nb_samples);

    av_frame_unref(av_frame);


    return true;
}

Here is how I would normally call this function:

VideoReader vr{};
if(!video_reader_open(&vr, "C:/Path/to/file.mp4"))
{
    Logger::error("Could not initialize VideoReader.");
    return 1;
}
unsigned char* buffer;
if(!video_reader_read_au_frame(&vr, &buffer))
{
    Logger::error("Could not read audio data.");
    return 1;
}

play_audio(&buffer);  <-- Find a way to play audio once buffer has data in it

video_reader_close(&vr);
return 0;

Obviously I will loop over video_reader_read_au_frame(&vr, &buffer) to playback the whole video.

I believe my code puts the samples from the decoded frame in buffer, but I am really not sure.. I am unsure as well if I need to convert to AV_SAMPLE_FMT_FLT audio format or something else or just leave it as it is. For libsoundio, I kind of understand this example: http://libsound.io/ but I'm not sure I fully understand how this library works, especially the callback function. I know I have to pass buffer in outstream->userdata as a void pointer, but I don't know how to use it in the callback function. Any help or guidance would be greatly appreciated. Note that later on in this project I might want to send this data over a network to play the video on another computer in sync.

Sound playback using FFmpeg and libsoundio in c++

0 Answers