Monitoring unnamed containers with cAdvisor and Prometheus

user_259453 0
Viewed 501

I have an experimental host running a Docker containerised app. cAdvisor and Prometheus containers are also running.

The app spawns containers to run jobs pertaining to one of a set of namespaces (could be thousands of namespaces) and names the container with a uuid. cAdvisor picks these up OK and I can see metric usage in prometheus OK per container. These containers run from a few seconds to about an hour.

I need to be able to correlate the namespace ID (from an API call in the container) with the metrics in prometheus as the container name is unique to the run and I need it by the namespace ID.

This doc says not to use lots of different labels as it generates lots of separate time series, but adding a kv seems the most convenient way.

This doc says to use the pushgateway for service-level batch jobs and as my containers are namespace-related rather than whole-service related, it seems incorrect to use this.

I might have 5-10 containers a day which run under different container names which correspond to the same namespace in the app, and I need a time series across all 5-10.

What is the best way to do this correlation?
How can I send a bit of arbitrary data from inside a container to prometheus?

cadvisor prometheus docker
1 Answers

If you need to differentiate a set of identical metrics by some characteristic, well there is no other reasonable way than to use a label. The problem with labels and storage arises when there are either too many labels or too many label values. Allow me to show you the problem on this example:

http_requests{code="200",client="8.8.8.8",uri="/hello"}

There are just three labels but each one of them can produce enormous amount of different time series. One per each unique combination of response code, client address and visited URL. Your namespace ID, I suppose, has a limited and small amount of possible values and therefore, wouldn't do any harm. It's even likely that unique container names (cAdvisor exposes them as labels) will have more impact than the namespace ID.

There is also a way to rewrite, add, or remove labels using relabel_config. Though, the solution is somewhat difficult to grasp and easy to shoot in the leg if you decide to rewrite. If the amount of possible namespaces is small I suggest proceed with adding a new label.

user_113445020
Related