We're trying to scale our Real Time Video Processing system to support over a hundred cameras. The system is majorly built in Python.
We're polling RTSP Camera streams using OpenCV and planning to deliver them using Kafka Producer. This part of the system is called Poller or Stream Producer.
The Cameras would be configured using a web interface and the Poller shall receive start/stop messages for any camera along with other details such as RTSP stream URL. This shall be done using Celery. For each start request, the Poller would create a new process for that camera and poll the stream using cv2.VideoCapture().read(). Captured frames would be sent over to Kafka tagged with camera ID and timestamped.
We're running all our components in Docker containers and intend to scale horizontally.
How can we scale Poller for a large (over a hundred or even more) number of cameras and effectively balance the camera streams across multiple instances of the Poller. Is there a way to achieve it using CPU/Memory metrics, or a more standard approach that we can follow for Docker.