Prometheus: how to get label_value() in query

user_6401395 0
Viewed 8562

So I have 2 metrics, I need to get label values from first metric and then query over 2nd metric, where {param="label_values_from_1st_metric"}. For ex:

metric_1{instance="abc"} - returns 3 timeseries:

metric_1{name="my-service", env="production", host="example-0.org"}

metric_1{name="my-service", env="production", host="example-1.org"}

metric_1{name="my-service", env="production", host="example-2.org"}

Next I need to make query over 2nd metric using host values from 1st query.

metric_2{domain="exmple-0,1,2.org"}

So the question is, how I can pass label_values to 2nd query? As far as I understand, I can use label_values() only for variables in grafana panel, so I can't write one query that will do that for me.

grafana prometheus promql metrics
3 Answers

You can try in the following way:

metric_2 * on(instance) group_left(host) metric_1{instance="abc"}

  • on(instance) => this is how to JOIN on label instance.

  • group_left(host) metric_1{instance="abc"} => means, keep the label host from metric_1 in the result.

user_148287900

Okay, finally, I've got it. So As Pulak Kanti suggested, I used group_left. However it returned error: many-to-many mathcing is not allowed. So what I've done:

  1. Labels between metric_1 and metric_2 do not match, however it is pretty simple to fix it, we need to add label_replace() func label_replace(metric_2,"host","$1","domain", "(.+)"). This expression adds to metric_2 label host with values copied from label domain. From our point of view we've just renamed label domain to label host. And this is great news, because now metric_1 and metric_2 have one common label - host. So the query from this step would look something like this:

    label_replace(metric_2,"host","$1","domain", "(.+)") + on (host) group_left(domain) metric_1{instance="abc"}

  2. Now our query at least doesn't retrun any error, however, it returns unexpected result-this is because it multiplies values from metric_2 with values from metric_1. Because we need only values from metric_2, we can just null metric_1, so it ends up being: label_replace(metric_2,"host","$1","domain", "(.+)") + on (host) group_left(domain) 0*metric_1{instance="abc"}

I do understand that this query isn't effective, however it worked for me. I would be very grateful if someone would help and make this query more efficient. BTW: here it is not important what is inside group_left() brackets, because it influences only what label we displays, for me it is not important, so label_replace(metric_2,"host","$1","domain", "(.+)") + on (host) group_left() 0*metric_1{instance="abc"} also works.

user_64013950

The following query may work faster if it is executed by optimized Prometheus-compatible query engine such as VictoriaMetrics:

metric_2 and on(domain) label_copy(metric_1{instance="abc"}, "host", "domain")

VictoriaMetrics starts from the right part of the a and b query, e.g. it executes label_copy(metric_1{instance="abc"}, "host", "domain"). Then it finds unique domain label values for the returned time series and passes an additional label filter on the domain label to the left part of the a and b query. Then it executes the left part of the query - metric_2{domain=~"domain1|...|domainN"} - and returns the matching time series.

Without this optimization all the time series with metric_2 name are selected before being dropped during the execution of a and b query according to these matching rules.

user_2749370
Related