I have a metrics set
session_duration_seconds{server="S.01",device_token="d0e025d0-ea09-4e7c-8ab1-6216d762a03f"} 2
session_duration_seconds{server="S.01",device_token="806c8c59-0f3c-4ec4-9538-f256408d4d74"} 8
session_duration_seconds{server="S.02",device_token="389c8c59-1336-2224-4538-8956404d4790"} 1
...many of them...
Generally speaking, for each server S.XX exposes a pile of metrics session_duration_seconds with unique device_token label value.
Those servers come from autodiscovery. So their names can vary. They can appear & disappear.
What I need to achieve:
- I need to calculate quantiles for every metric with the same server label. (Ideally, I will put that into the Recoding Rule for auto-recalculation.) But how to write such a PromQL query when must get labels from a sub-query?
- Create an alert for a situation when a server stops exposing session_duration_seconds metric. Like absent_over_time for every server label. But the server (target) list is constantly changing.