Filter vector based new label introduced by `label_replace`

user_9257739 0
Viewed 1543

In Prometheus I'm trying to merge multiple http request lines into groups using label_replace.

http_requests_total{account_id="124",handler="AAAAAController"...}
http_requests_total{account_id="125",handler="BBBBBController"...}
http_requests_total{account_id="126",handler="CCCCCController"...}
http_requests_total{account_id="123",handler="XXXXXController"...}

The query I wrote is :

label_replace(http_requests_total, "class", "$1", "handler", "([a-zA-Z0-9]+)Controller.*") .

This works correctly and adds the class label to vector :"AAAA","BBBB" etc. At this point I would like to remove certain classes such as empty and BBBB.

How can I further filter the vector using {class~="BBBBB"} :

label_replace(http_requests_total, "class", "", "handler", "([a-zA-Z0-9]+)Controller.*"){class~="BBBBB"}

Prometheus shows an error when I try to do so.

grafana prometheus promql
3 Answers

I've noticed that there is a grafana tag under this question, so if you don't mind to use grafana to achive that goal, then there is another way.

By using Grafana... 7.x+ IIRC, you can use Transform to reprocess the raw result set.

For instance, I have a Query result like this:

enter image description here

Add a Filter data by values plugin. You can set Match any in it, and add a regex condition. Set field Class matches BBBBB or anything you want.

enter image description here

user_88840130

I think the record rules feature might be helpful to you. Following is the steps for your reference

  1. create record rule. Prometheus allow us to set 2 type rules, one is alert rule another one is record rule. Record rule will create new metrics by the setting that you provide on Prometheus server setting. Here is the sample according your description.

groups:
- name: http_requests_total_new_merics
  rules:
  - record: http_requests_total_new
    expr: label_replace(http_requests_total, "class", "", "handler", "([a-zA-Z0-9]+)Controller.*")

then Prometheus will create new metrics named http_requests_total_new

  1. Using PromQL to query according your needs, e.g.
  • http_requests_total_new{class!="BBBBB"}
user_176095640

The filtering for this particular question can be done via the inner series selector before applying label_replace. For example, the following query returns only series with class=~"BBBBB":

label_replace(
  http_requests_total{handler=~"BBBBBController.*"},
  "class",
  "$1",
  "handler",
  "([a-zA-Z0-9]+)Controller.*"
)

P.S. While Prometheus doesn't provide the ability to post-filter time series by label values after they are selected by series selector, this can be done in VictoriaMetrics - Prometheus-like system I work on. See label_match and label_mismatch functions.

user_2749370
Related