Alerts with several expressions

Viewed 217

I'm a thinking of a concept in terms how to define alerts based on latency. Essentially I would need that for the blackbox exporter alerts. Based on the reponse time I won't have alert stating the current status of the connection.

good connection
slow connection
no connection

Is there a way to define one rule which will evalute more then 1 expression and fire an alarm with specific summary and labels depending on the expression that fires?

1 Answers

The answer is rather 'no' than 'yes'. Annotations are static, whatever you wrote in it will be there for each trigger. You can, however, execute queries in annotations, but I guess bringing there any logic would be an overkill. To show you the complexity, consider this annotation:

# get error codes that occurred during last 5 minutes in the firing service
error_codes: >-
  Error code(s): {{ range printf `sum(increase(request_duration_seconds_count{status=~"5..",hostname="%s",service="%s"}[5m])) by (status) > 0` .Labels.hostname .Labels.service | query | sortByLabel "status" }}{{ .Labels.status }} {{ end }}

The situation with labels is somewhat the same. Those labels you write in alert under labels: key are static. But you can mess with labels in alert expression by joining, rewriting, adding new, etc. Just as with annotations, this will add a lot of unreasonable complexity, which will make you unsure whether you actually get an alert or there is bug that will let you down under some circumstances.

My advice is to Keep It Simple, Stupid. Make several alerts but let them be easy to understand, customise, and fix.

Related