OpenGL Off-screen Rendering flickering image (OSMesaMakeCurrent )

I have been trying to solve a bug with OpenGL where my image flickers

Here is a snippet that summarizes the issue and where I think the issue lies

std::vector<uint8_t> buffer;
buffer.resize(100 * 100 * 4);
std::vector<uint8_t> buffer_2;
buffer_2.resize(100 * 100 * 4);

OSMesaMakeCurrent(ctx, buffer.data(), GL_UNSIGNED_BYTE, 100, 100);

while (true)
{
  glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);
  glFinish();

  // draw_frame basically sets up and draws a 2D image based on some conditions
  const bool new_frame = draw_frame();

  // This is my way of "double buffering"
  // Basically, if drawing the 2D image was successful
  // copy it into buffer 2
  // This is done because frames are sometimes not available but text must keep moving
  if (new_frame)
  {
    glFinish();
    
    buffer_2 = buffer;
  }
  else
  {
    // If it can't draw, place the old image
    buffer = buffer_2;
  }
   
  // I printed the pixels at this point and they look like they are there

  // This basically draws some text on top of the image
  draw_text();
  glFinish();
 
  // I printed the pixels at this point, they look good when new_frame is true, but when false you could tell there is empty space (no image)

  // After this I simply make a copy of the image, and send it to a display on another app
}

When draw_frame() returns true, I can see the image, but when it returns false, I can only see the text

This code worked with OpenGL 2.1 (can't remember the mesa version)

But does not seem to work on OpenGL 3.1 Mesa 19.3.1

It seems to ignore the pixel values that are written in the else

  else
  {
    // If it can't draw, place the old image
    buffer = buffer_2;
  }

Something must have changed between versions but I can't seem to figure out what, I have tried

1. Moving the glClear to different places
2. Adding glFlush and glFinish to some other places
3. Moving glClear to only when I draw video (This causes the text to write on top of itself becoming unreadable after a while)