Instance label rewrite when __address__ is the same and Prometheus is a sidecar container

user_666259 0
Viewed 210

I have a setup in AWS Lightsail Containers where Prometheus is loaded as a sidecar container for every app container. So if I have two app containers, I also have two Prometheus containers running. (The Prometheus containers forward metrics using remote_write to a global Prometheus instance that's used for querying.)

The Prometheus configuration for every sidecar container looks like this:

global:
  scrape_interval:     15s

scrape_configs:
  - job_name: app
    static_configs:
      - targets:
          - localhost:8080

Note that the target address is the same for every target.

As I understand it, the instance label should be unique for every target (?), but here it's not. How could I use label rewrites to change them to something unique? Do I have e.g. the IP address available as a meta label? I'm not using any service discovery, so e.g. __meta_lightsail_instance_name is not available.

prometheus
1 Answers

I ended up solving this by using external labels with environment variables in my prometheus.yml global config:

global:
  external_labels:
    container: ${HOSTNAME}

The ${HOSTNAME} env var is set by ECS Fargate / Lightsail Containers on the container.

For this to work, a feature flag needs to be enabled, called --enable-feature=expand-external-labels. My final Dockerfile looks like this:

FROM prom/prometheus
ADD prometheus.yml /etc/prometheus/
ENTRYPOINT ["/bin/prometheus"]
CMD ["--config.file=/etc/prometheus/prometheus.yml", \
     "--storage.tsdb.path=/prometheus", \
     "--storage.tsdb.retention.time=1d", \
     "--enable-feature=expand-external-labels"]
user_6662590
Related