Prometheus: Prevent starting new time series on label change

user_3737935 0
Viewed 681

Assuming the following metric:

cpu_count{machine="srv1", owner="Alice", department="ops"} 8
cpu_count{machine="srv1", owner="Bob", department="ops"} 8

I'd like to be able to prevent starting a new time series on owner change. It should still be considered the same instance, but I would like to be able to look up by owner. I don't particularly care if it matches only on my_metric{owner=~"Box"} or on both my_metric{owner=~"Box"} and my_metric{owner=~"Alice"}, I just need to make sure it does not count twice on my_metric{machine=~"srv1"} or my_metric{department=~"ops"}.

I'm willing to accept that using labels to group instances in this manner is not the correct approach, but what is?

prometheus label
2 Answers

When you add the label "owner" to this kind of metric I think you're trying to accomplish a kind of "asset management" which could be done better with some other tool developed specific to this goal. Prometheus isn't a suitable tool to keep the information of who is using each machine in your company.

Said that, every time the owner of a machine changes you could workaround this issue deleting the old data series using the REST API executing something like this:

curl --silent --user USER:PASS --globoff --request POST "https://PROMETHEUS-SERVER/api/v1/admin/tsdb/delete_series?match[]={machine='srv1',owner='Bob'}"
user_46536750

If you can change the code, it would be better to have a metric dedicated to the ownership:

 # all metrics are identified a usual
 cpu_count{machine="srv1", department="ops"} 8
 # use an info metrics to give details about owner
 machine_info{machine="srv1", owner="Alice", department="ops"} 1

You can still aggregate the information id you need it:

cpu_count * ON(machine,department) machine_info

That way, the owner is not polluting all your metrics. Still, you will have issues when changing the owner of a machine while waiting for the older metric to disappear (5 minutes before staleness).

I have not tried it but a solution could be to use the time at which the ownership changed (if you can provide it) as a metric value - epoch time in seconds.

 # owner changed at Sun, 08 Mar 2020 22:05:53 GMT
 machine_info{machine="srv1", owner="Alice", department="ops"} 1583705153
 # Previous owner Sat, 01 Feb 2020 00:00:00 GMT
 machine_info{machine="srv1", owner="Alice", department="ops"} 1580515200

And then use the following expression to get the latest owner whenever you need the current owner - only useful when owner has changed within the last 5 minutes:

machine_info == ON(machine,department) BOOL (max(machine_info) BY(machine,department) )

Quite a mouthful but it would approach what you want.

user_34808080
Related