Our Prometheus setup is that we have 1 app metrics that ended up as 2 prometheus metrics with different label prometheus_replica. I believe they are using k8s prometheus operator to set it up.
What I found out there are a lot of missing values randomly between these 2 metrics. Some time same data point are in both metrics replica, and some time only one of them.
example:
replica-0
replica-1
But if I put them together, i see the whole data
My question is:
- Is this a normal setup or just the infrastructure behind it is just bad?
- If This is normal. How would I construct the query. For example if I want to get the sum of a single counter over 5 minutes using both metrics.
- I assume that I should be able to use only just one of the replica for my query. is that correct?