I have an histogram of requests with path and status codes...How can I alert if the errors are increasing with 20% in the last hour compared with the hour before?
one metric sample:
{instance="someIp",instance_hostname="someHost",job="someAppName",le="+Inf",method="GET",path="somePath",status_code="500"}
I should rely on rate function? something like:
rate(http_request_duration_seconds{job="someProject", status_code="500"}[60m])
Take the ratio of the error rate over the past hour to the error rate over the previous 1 hour:
(
rate(http_request_duration_seconds_count{status_code="500"}[1h])
/
rate(http_request_duration_seconds_count{status_code="500"}[1h] offset 1h)
)
>
1.2
This will check for an increase in the absolute number of errors. If you want to check for an increase in the relative number of errors (e.g. 10% of requests failed over the past hour vs 5% over the previous hour), then you'd need to divide the error rate by total request rate before comparing to the same thing 1 hour ago.
Or you may want to combine the two, and alert if the relative number of errors has increased by X%, and the absolute number is over some noise threshold (so that it doesn't trigger if you got 2 requests over the past hour and one of them failed).