What is the optimal way to synchronize frames in ffmpeg c/c++?

Turgut 0
Viewed 55

I made a program that read's n number of video's as input, draws those videos to the GLFW window and finally encodes it all as a singular video output. The problem is frames of each video in question can be different, it's dependent on the user's input.

For example: the user can put two video's which has an FPS of 30 and 59, and can want an output 23,797. The problem is those video's are not in sync with each other, thus on the output we can see that the input video's are either faster or slower.

Duration of each video is also dependent on the input. For example, in accordance to the previous example, the first input might be 30 second and the second can be 13 second, while the output is 50 seconds.

I mostly read the frames similar to a moving png rather than a solid video since there are no iframe and bframes. There are just data I get from the GLFW window.

As an example, let's say we give one video as input which has an FPS of 30 and duration of 30, and our output has an FPS of 23.797 and duration of 30. I have 2 function's skip_frame and wait_frame which respectively either read's a frame twice so we skip a frame or don't read the frame on that iteration. Those function's are used depending on the situation, whether it's output < input or output > input.

Here is what my code roughly looks like:

while(current_time < output_duration){
   for(auto input_video: all_inputs){
      for(int i = 0; i < amount_to_read_from_input(); i++){
         frame = input_video.read_frame();
      }
   }
   
   GLFW_window.draw_to_screen(frame);

   encoder.encode_one_video_frame(GLFW_window.read_window());
}

Basically skip_frame and wait_frame are both inside amount_to_read_from_input() and return 2 or 0 respectively.

So far I have tried multiplying duration with fps for both input and output. Then getting the result of their subtraction. Going from our previous example we get 900 - 714 = 186. Then I divide the result to the output fps like so: 714 / 186 = 3.8. Meaning that I have to skip a frame every 3.8 iterations. (I skip a frame every 3 iterations and save the residual 0.8 for the next iter.)

But it's still a seconds or two behind. (Like it ends at 29 seconds for a 30 second output.) and the audio is out-of-sync. Ffmpeg handles my audio so there are no errors on that part.

Also seen this question but I don't think I can utilize ffmpeg's function's here since im reading from a glfw window and it comes down to my algorithm.

The problem is what is the math here?

What can I do to make sure these frames are stabilized on almost every input/output combination?

ffmpeg video c c++
1 Answers

For those who are struggling in a similar situation: The problem was my encoder's time base: I was passing it as an int rather than a formatted AVRational. My fps variable was a float, and when I pass that to my encoder it rounds it to the closest integer. As an example I was passing 23.797 and it rounded to 23. So whatever algorithm I write it was always wrong since my algorithms were wired to calculate for 23.797 while ffmpeg was trying to encode for 23.

Changing time_base = (AVRational){ 1, fps}; to time_base = av_d2q(1 / STREAM_FRAME_RATE, std::numeric_limits<int>::max()); caused all my algorithm's to work as intended.

I'm still not sure whether there is a standard for calculating this or not, my way of doing it worked for me just fine.

Turgut0
Related