Unit test for z-score with Prometheus

user_397940 0
Viewed 80

I have been experimenting a lot with writing unit tests for alerts as per this: https://prometheus.io/docs/prometheus/latest/configuration/unit_testing_rules/#alerts-yml I have some simple cases out, but now I am tackling rules that are less trivial. For example this:

abs(
  avg_over_time(my_metrics{service_name="aService"}[1m])
  - 
  avg_over_time(my_metrics{service_name="aService"}[3m])
)
/ stddev_over_time(my_metrics{service_name="aService"}[3m])
> 3

I have one file with the above rule and then this is in my test:

  - interval: 1m
      # Series data.
    input_series:
      - series: 'my_metrics{service_name="aService"}'
        values: '0 0 0 0 1 0 0 0 0 '
    alert_rule_test:
      - eval_time: 3m
        alertname: myalert
        exp_alerts:
          - exp_labels:
              severity: warning
              service_name: aService
            exp_annotations:
                summary: "some text"
                description: "some other text"

I am not sure what my series should look like in order to test deviation from the mean. Is it even possible to test such rule?

Thank you

EDIT I can have a succesful test if I set it > 0 as opposed to >3 I have tried to set a series of this sort:

'10+10x2 30+1000x1000'

but I cannot understand what would be the correct setup to have it triggered

prometheus alerts unit-testing
1 Answers

This isn't a direct answer, rather a tip from someone who spent quite some time on these tests. Did you know that apart from testing alert expressions, you can unittest PromQL expressions as well? See how it can be useful:

evaluation_interval: 1m
tests:
- interval: 1m
  input_series:
  - series: test_metric
    values: 1 1 1 10 1 1 1

  promql_expr_test:
    - expr: avg_over_time(test_metric[1m])
      eval_time: 4m
      exp_samples:
        -  value: #5.5
    - expr: avg_over_time(test_metric[3m])
      eval_time: 4m
      exp_samples:
        - value: #3.25
    - expr: stddev_over_time(test_metric[3m])
      eval_time: 4m
      exp_samples:
        - value: #3.897114317029974

I've split your alert expression into three separate, simple parts. If you run this unittest, you will see the commented-out values in the error message. From here it is not difficult to join pieces together and see why the alert is not happening. You can use that to build a working sequence of values.

user_113445020
Related