Traefik V2.0 Metrics with Prometheus

user_3726059 0
Viewed 11889

If I want to capture metrics of Traefik with Prometheus, this only works if insecure= true is activated. If I set insecure=false I get a 404. Maybe someone has an example of what a correct configuration looks like. So the configuration of Traefik (V2.0 CLI) and the prometheus.yml.

traefik.yml command Section:

      command:
        - "--metrics=true"
        - "--metrics.prometheus=true"
        - "--metrics.prometheus.buckets=0.100000, 0.300000, 1.200000, 5.000000"
        - "--metrics.prometheus.entrypoint='metrics'"
        - "--metrics.prometheus.addEntryPointsLabels=true"
        - "--metrics.prometheus.addServicesLabels=true"

        - "--api=true"
        - "--api.dashboard=true"

        - "--log.level=INFO"

        - "--providers.docker=true"
        - "--providers.docker.swarmmode=true"
        - "--providers.docker.exposedbydefault=false"

        - "--entrypoints.web.address=:80"
        - "--entrypoints.websecure.address=:443"
        - "--certificatesResolvers.sec.acme.email=foo@bar.com"
        - "--certificatesResolvers.sec.acme.storage=/letsencrypt/acme.json"
        - "--certificatesResolvers.sec.acme.httpChallenge.entryPoint=web"
      deploy:
        labels:
            - "traefik.enable=true"

            - "traefik.http.routers.api.rule=PathPrefix(`/api`) || PathPrefix(`/dashboard`)"
            - "traefik.http.routers.api.rule=Host(`foo.bar`)"

            - "traefik.http.routers.api.service=api@internal"
            - "traefik.http.routers.api.middlewares=myauth"
            - "traefik.http.services.api.loadbalancer.server.port=8080"
            - "traefik.http.routers.api.tls.certresolver=sec"
            - "traefik.http.middlewares.myauth.basicauth.users=xxx"

prometheus.yml:

global:
    scrape_interval: 10s
    scrape_timeout: 10s
scrape_configs:
    - job_name: 'pushgateway'
      static_configs:
              - targets: ['pushgateway:9091']
      honor_labels: true
    - job_name: 'traefik'
      scheme: https
      static_configs:
              - targets: ['foo.bar']
      basic_auth:
              username: myusername
              password: mypassword
prometheus traefik metrics
3 Answers

First expose 8082 port if you're in docker.

And add these commands when starting Traefik.

--metrics.prometheus=true
--metrics.prometheus.entryPoint=metrics
--entryPoints.metrics.address=:8082

Access that metrics in domain:8082/metrics

If you also add the following labels it will allow you to access the metrics on traefikmetrics.mydomain.com/metrics. You need to fill in the values in the angle braces depending on your setup.

traefik.http.routers.traefikmetrics.entrypoints: <http or https entrypoint>
traefik.http.routers.traefikmetrics.rule: Host(`traefikmetrics.mydomain.com`)
traefik.http.services.<traefik-service-name>.loadbalancer.server.port: 8082
user_47975770

I have a working configuration for a prometheus metrics endpoint behind the Traefik reverse proxy itself without using the insecure configuration (although using a toml configuration file) - not quite sure, if that is what you want to achieve.

So from your example configuration above, from what I understand, you are missing some stuff.

First you configured the prometheus metrics to use the entrypoint 'metrics', so you need to configure an entrypoint called metrics in the CLI section, e.g. like this:

 - "--entrypoints.metrics.address=:8082"

If you want to access the metrics endpoint via TLS (terminated by traefik itself) and use basic authentication, you need a router configuration and service for the metrics endpoint as labels, e.g. like the following (additionally to your configuration of the api endpoint):

   - "traefik.http.routers.metrics.rule=PathPrefix(`/metrics`)"
   - "traefik.http.routers.metrics.rule=Host(`foo.bar`)"
   - "traefik.http.routers.metrics.tls=true"
   - "traefik.http.routers.metrics.tls.certResolver=sec"
   - "traefik.http.routers.metrics.service=metrics"
   - "traefik.http.routers.metrics.middlewares=myauth"
   - "traefik.http.services.metrics.loadbalancer.server.port=8082"

Additionally I needed to configure the docker swarm network like this (otherwise I am getting a 503 Gateway timeout):

- "traefik.docker.network=proxy"

(proxy is my overlay network connecting traefik and all services which Traefik will be the reverse proxy for.)

Hope that helps...

user_123172770

If you want to access the api and dashboard in secure mode then you will need to use the api@internal 'magic' explained in the Traefik documentation here.

This means adding a label like:

"traefik.http.routers.<my-api-dash-name>.service=api@internal"

Needless to say this is not cool. In essence we are treating this as a special snowflake situation, as opposed to using standard Traefik service declarations. It appears that this is somewhat acknowledged, and future Traefik versions plan at a minimum to make this internal service appear in the dashboard.

Note that when you use insecure mode for the dashboard it opens up port 8080 on your local Traefik server. When you switch to secure 8080 is closed.

I, like you, sought to declare my dashboard/api securely by routing internally to 8080. If you are doing so via https and some auth mechanism, and not even exposing 8080 as a docker service port then arguably IMO you could be considered secure - but you will need to leave the insecure dashboard switch in your Traefik config so that you can leverage 8080 in this way.

If others have views on the security model described in prior paragraph vs. the cryptic api@internal I am all ears.

user_95354630
Related