We are looking to implement a monitoring and alerting solution where we would like to give every functional unit there own prometheus instance.
Currently we are running it via prometheus-operator with single prometheus instance but now we need to scale it to multiple prometheus instances with a single operator.
Can someone please provide me to the right direction?
This is quite straightforward: just deploy multiple Prometheus objects to your cluster with different configurations. prometheus-operator will manage any number of prometheus instances.
Added after comment clarification:
prometheus-operator is a piece of software running in your cluster that looks for new/changed/deleted objects with kind: Prometheus (and some others) and creates "regular" k8s objects based on Prometheus entities config.
You can absolutely install multiple instances of prometheus-operator chart (I'm assuming you mean chart that was named promethues-operator and now is renamed to https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)
At any time there should be only one instance of prometheus-operator (software, not chart) running, so just make sure that any additional releases do not install prometheus-operator itself (software, not the chart). It is available as a values parameter to pass to helm:
prometheusOperator:
enabled: false
We faced similar issues while scaling up the prometheus infrastructure. However prometheus is not designed for the scalability. Basically you create one instance of prometheus and do everything with it.
What you can do is to deploy the Cortex on top of prometheus. Its horizontally scalable storage.
Then you can add configuration in cortex to scrape the metrics. (similarly whatever you are doing right now) https://github.com/cortexproject/cortex