How to write a Prometheus query that returns result for every label value?

Viewed 160

I have a metrics set

session_duration_seconds{server="S.01",device_token="d0e025d0-ea09-4e7c-8ab1-6216d762a03f"} 2
session_duration_seconds{server="S.01",device_token="806c8c59-0f3c-4ec4-9538-f256408d4d74"} 8
session_duration_seconds{server="S.02",device_token="389c8c59-1336-2224-4538-8956404d4790"} 1
...many of them...

Generally speaking, for each server S.XX exposes a pile of metrics session_duration_seconds with unique device_token label value.

Those servers come from autodiscovery. So their names can vary. They can appear & disappear.

What I need to achieve:

  1. I need to calculate quantiles for every metric with the same server label. (Ideally, I will put that into the Recoding Rule for auto-recalculation.) But how to write such a PromQL query when must get labels from a sub-query?
  2. Create an alert for a situation when a server stops exposing session_duration_seconds metric. Like absent_over_time for every server label. But the server (target) list is constantly changing.
0 Answers
Related