Prometheus show top metrics filtered by number of labels

Viewed 4591
1 Answers

Prometheus provides /api/v1/status/tsdb endpoint, which exposes metric names with the highest number of labels inside seriesCountByMetricName stats. It also exposes other useful stats, which may help determining the source of high cardinality:

  • labels with the highest number of unique values at labelValueCountByLabelName stats
  • label=value pairs with the highest number of unique time series at seriesCountByLabelPair stats

P.S. Other Prometheus-like systems may provide additional functionality for /api/v1/status/tsdb endpoint, which can help narrowing down the source of high cardinality. For example, VictoriaMetrics provides the following additional functionality:

  • Ability to specify the date for the stats via date query arg. For example, /api/v1/status/tsdb?date=2022-04-01 would return stats for April 1, 2022.
  • Ability to specify the number of entries to return per each collected stats via topN query arg. For example, /api/v1/status/tsdb?topN=100 would return up to 100 entries per each stats list.
  • Ability to filter out time series for stats collection via arbitrary time series selector. For example, /api/v1/status/tsdb?match[]=foobar would return stats for time series with foobar name only.

See these docs for more info.

Related