How dangerous are high-cardinality labels in Prometheus?

user_90328 0
Viewed 15477

I'm considering exporting some metrics to Prometheus, and I'm getting nervous about what I'm planning to do.

My system consists of a workflow engine, and I'd like to track some metrics for each step in the workflow. This seems reasonable, with a gauge metric called wfengine_step_duration_seconds. My issue is that there are many thousands of steps across all my workflows.

According to the documentation here, I'm not supposed to programmatically generate any part of the name. That precludes, then, the use of names such as wfengine_step1_duration_seconds and wfengine_step2_duration_seconds, because the step names are programmatic (they change from time to time).

The solution, then is a label for the step names. This also presents a problem, though, because the documentation here and here cautions quite strongly against using labels with high cardinality. Specifically, they recommend keeping "the cardinality of your metrics below 10", and for cardinality over 100, "investigate alternate solutions such as reducing the number of dimensions or moving the analysis away from monitoring".

I'm looking at a number of label values in the low thousands (1,000 to 10,000). Given that the number of metrics otherwise won't be extremely large, is this an appropriate usage of Prometheus, or should I limit myself to more generic metrics, such as a single aggregated step duration instead of individual duration for each step?

prometheus
2 Answers

High-cardinality labels (e.g. labels with big number of unique values) aren't dangerous on their own. The danger is in the total number of active time series. A single Prometheus instance can handle up to ten millions of active time series according to https://www.robustperception.io/why-does-prometheus-use-so-much-ram when running on a host with >100GB of RAM.

An example: suppose the exported metric has a step_id label with 10K unique values.

If the metric has no other labels (e.g. if it is exported as wfengine_duration_seconds{step_id="...}), then it will generate 10K active time series (tiny value for Prometheus).

If the metric contains another label such as workflow_id with 100 unique values and each workflow has 10K unique steps, then the total number of exported time series skyrockets to 100*10K=1M. This is still pretty low number of active time series for Prometheus.

Now suppose that the app, which exports the metric, runs on 50 hosts (or Kubernetes pods). Prometheus stores the scrape target address in the instance label - see these docs. This means that the total number of active time series collected from 50 hosts jumps to 50*1M=50M. This number may be too big for a single Prometheus instance. There are other systems, which can handle such amount of active time series in a single-node setup, but they also have upper limit. It is just N times bigger (1 < N < 10).

So the rule of thumb is to take into account the number of active time series, not the number of unique values per a single label.

user_2749370

The guideline of staying under 100 cardinality for your biggest metrics presumes that you have 1000 replicas of your service, as that's a reasonably safe upper bound. If you know that everyone using this code will always have a lower number of replicas, then there's scope to have a higher cardinality in instrumentation.

Saying that, thousands of labels is still something to be careful with. If it's already tens of thousands, how long before it's hundreds of thousands? Long term you'll likely have to move this data to logs given the cardinality, so you may wish to do so now.

user_835910
Related