OpenGL slows for 5k points

I am writing a SLAM library and want to visualize its work with OpenGL. I need to draw some 100k points and a few hundred rectangles and I would expect that OpenGL can handle it easily. However, after the number of points reaches 5k my program slows down.

I am new to OpenGL, so I guess I am not doing things in a proper way. I used this tutorial for learning.

As the program works, it spits out certain events from which only a few are relevant:

• Point Created (size_t id, float x,y,z;)
• Point Updated (szie_t id, float x,y,z;)
• Position Estimated (Quaternion Q, Vector3D T)

The part of the program that visualizes these events (simplified) works as follows.

We assign a GL_ARRAY_BUFFER to each point. In order not to allocate a new buffer every time we get a new point, I decided to keep a repository of buffers. As a new point arrives we assign to it a "free" buffer from the repository. Only if the repo is empty, we allocate a new buffer with glGenBuffers.

std::stack<GLuint> point_buffer_stack;
std::map<size_t, GLuint> active_points;

void OnPointCreated(size_t id,  float x, float y, float z ){
  GLuint point_buffer_id;
  if(point_buffer_stack.empty()){
    glGenBuffers(1, &point_buffer_id);
  }
  else {
    point_buffer_id = point_buffer_stack.top();
    point_buffer_stack.pop(); 
  }
  active_points[id] = point_buffer_id;

  OnPointUpdated(id, x, y, z);
     
}

void OnPointUpdated(size_t id, float x, float y, float z){
  GLuint point_buffer_id = active_points[id];
  float buffer[3] = {x,y,z};
  glBindBuffer(GL_ARRAY_BUFFER, point_buffer_id);
  glBufferData(GL_ARRAY_BUFFER, sizeof(buffer), buffer, GL_STATIC_DRAW); 
}

void OnPointDeleted(size_t id){
  GLuint point_buffer_id = active_points[id];
  point_buffer_stack.push(point_buffer_id);
  active_points.erase(id);
}
  

Draw the frame only when the position is updated:

void OnPositionUpdated (const glm::mat4 & projection){
  glm::mat4 model_view_projection;
  /* Compute  model_view_projection and set the corresponding UniformMatrix4fv for using in the  vertex shader */

  // Draw points 
  glEnableVertexAttribArray(0);
  for(const auto & id_vertex: active_points){
    glBindBuffer(GL_ARRAY_BUFFER, id_vertex.second);
    glVertexAttribPointer(
      0,                  // layout from vertex shader.
      3,                  // 3D
      GL_FLOAT,           // type
      GL_FALSE,           // normalized?
      0,                  // stride
      (void *) 0            // array buffer offset
      );
    glDrawArrays(GL_POINTS, 0, 1);
  }
  glDisableVertexAttribArray(0);

  /*Draw a rectangle that shows the current position by drawing two triangles */

  glfwSwapBuffers(window);
  glfwPollEvents();
}

On my Asus Zenbook with Intel® Xe Graphics, Ubuntu 20.04, OpenGL starts to lag behind the camera at ~5k points. After some time I get an insufficient memory error from OpenCV (which is built with OpenGL support).

Is this expected? What am I doing wrong? How to solve this issue?

For the reference, the initialization of OpenGL is done as follows:

glfwInit();
glfwWindowHint(GLFW_SAMPLES, 4);
glfwWindowHint(GLFW_DOUBLEBUFFER, GLFW_TRUE);
glfwWindowHint(GLFW_CONTEXT_VERSION_MAJOR, 3);
glfwWindowHint(GLFW_CONTEXT_VERSION_MINOR, 3);
glfwWindowHint(GLFW_OPENGL_FORWARD_COMPAT, GL_TRUE); // To make MacOS happy; should not be needed
glfwWindowHint(GLFW_OPENGL_PROFILE, GLFW_OPENGL_CORE_PROFILE);

window = glfwCreateWindow(1920, 1080, "SLAM 3D visualization", NULL, NULL);