We have a metric, let's say http_requests_total. This is generated by multiple pods in a Kubernetes cluster for the one service. There are also multiple clusters around the world generating the same metric. This also has a custom label called customer_id which lets us group the metric by each customer. This is scraped per minute.
We want to see this aggregated across all pods in the service & federated to a central Prometheus, so we have a rule in each Prometheus that generates a new metric called http_requests_total_rate which is the result of sum by(customer_id) (rate(http_requests_total[2m])). This rule metric is what we then scrape into the federation Prometheus.
So now we have a nicely rated then summed metric that we can view per-minute global HTTP request counts per minute, per customer by querying sum by(customer_id) (http_requests_total_rate).
The issue is, how can I see a per-day metric from this federated metric? I can't do any of the normal tricks like increase(http_requests_total[24h]) or sum_over_time because these require a range from a counter, and we already have a rate.
Can this be done with the federated metric that is already a rate?