How to filter by two labels in prometheus?

user_6931919 0
Viewed 1455

We have an error_count metric which have two labels:

  1. client
  2. error_code

Because we know that there is a client will have (let's say, 404 error_code) which can not be avoided, so we want to exclude it in our dashboard.

Something like:

error_count{client!="foo_client" and error_code!="404"}

But

error_count{client!="foo_client", error_code!="404"}

won't work because it will exclude all from foo_client and all 404 errors.

And for what's worse, I only have control on Grafana's PromQL, maybe config Prometheus' collecting can solve this but I have no permission to config this since they are managed by other teams.

grafana prometheus promql monitor
2 Answers

If I understand correctly, what you want is the error_count metric for all clients. If it's a foo_client, you only want it if it's not a 404 error code.

Here's one way to do that in Prometheus: get all entries that aren't foo_client, OR entries that are foo_client that don't have error_code=404:

error_count{client!="foo_client"} or error_count{client="foo_client", error_code!="404"}
user_28293810

You can use the or which is an union operator in promQL.

error_count{client!="foo_client"}
or
error_count{error_code!="404"}
user_20969860
Related