How to alert on JVM memory usage in Prometheus with Micrometer and Alertmanager

user_2497239 0
Viewed 12125

I am new to Prometheus and Micrometer. I am trying to alert when the heap memory usage of the JVM is exceeding a certain treshold. 

- alert: P1 - Percentage of heap memory usage on environment more than 3% for 5 minutes.
    expr: sum(jvm_memory_used_bytes{application="x", area="heap"})*100/sum(jvm_memory_max_bytes{application="x", area="heap"}) by (instance) > 3
    for: 5m
    labels:
      priority: P1
      tags: infrastructure, jvm, memory
    annotations:
      summary: "Percentage of heap memory is more than threshold"
      description: "Percentage of heap memory for instance '{{ $labels.instance }}' has been more than 3% ({{ $value }}) for 5 minutes."

Now this expression is working when I use this on Grafana:

Grafana example

But in Prometheus it looks like this:

Query in Prometheus

How can make my alerts to alert when the memory usage goes above a certain limit?

micrometer prometheus alert jvm
2 Answers

You want to average the heap usage over time. I came up with the following:

- name: jvm
  rules:
    - alert: jvm_heap_warning
      expr: sum(avg_over_time(jvm_memory_used_bytes{area="heap"}[1m]))by(application,instance)*100/sum(avg_over_time(jvm_memory_max_bytes{area="heap"}[1m]))by(application,instance) >= 80
        for: 1m
        labels:
          severity: warning
        annotations:
          summary: "JVM heap warning"
          description: "JVM heap of instance `{{$labels.instance}}` from application `{{$labels.application}}` is above 80% for one minute. (current=`{{$value}}%`)"
user_90715020

Your alert is correctly configured to only alert when the result of the query is above 3 for 5 straight minutes. Based on the graph in Prometheus of the query, it hasn't done that in the past hour, so no alert is being generated.

It's worth noting, also, that the query you're using for the rule will only return the instance label of each result. So, if you were planning on using the application label in your alert, you will need to either adjust the query to return the application label as well, or add that label to the list of labels that get added in the rule.

user_95442570
Related