OpenGL and ffmpeg make video with stable fps

I've made a program that takes multiple vidoes as inputs, have ffmpeg decode them, send them to opengl, then create a window using glfw, draw textures on the screen using those videos (Edits those textures), then I read the screen using glReadPixels so ffmpeg can encode it. I send the read frames to the encoder and it encodes it. I specify the fps on start, but the problem is the video is faster then it's supposed to be. Now I can do something like this:

double pt_in_seconds = pts * (double)time_base.num / (double)time_base.den;
while (pt_in_seconds > glfwGetTime()) {
      glfwWaitEventsTimeout(pt_in_seconds - glfwGetTime());
}

But the problem with this is that this approach makes the run-time really long. So if I input a 1 hour video I have to wait for 1 hours. If I don't use this code snippet it generates the output as fast as it can, but like I said the output video is faster than it's supposed to be. Whats shown in the glfw window is irrelevant, it's hidden anyways, it's just there to manipulate/merge input videos.

Is there a better way for ffmpeg to stabilize the encoded information? At the end of the day glfw just displays the decoded videos, since they are both on the same iteration.

It looks roughly like this:

...
while(true)
{
   // The actual program originally reads every input inside a vector here.
   // But since the program itself is really long I just did this as a representation
   uint8_t* decoded_data = decoder.decode_one_frame();
   
   // draw_frame_on_screen returns glReadPixels result.
   uint8_t* screen_data = opengl_engine.draw_frame_on_screen(decoded_data);

   encoder.encode_one_frame(screen_data);
}

Encoder is entirely just muxing.c from ffmpegs official docs, I've just removed the dummy image and added my screen_data as input.

Using ubuntu, GLFW, GLAD, ffmpeg.