Custom metrics is not showing in prometheus web ui so does in grafana

user_8502433 0
Viewed 101

First of all I tried this solution didn't work for me.

I need to log some custom metrics using Prometheus.

docker-compose.yml

version: "3"
volumes:
  prometheus_data: {}
  grafana_data: {}
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    hostname: my_service
    ports:
      - 9090:9090
    depends_on:
      - my_service
  my-service:
    build: .
    ports:
      - 8080:8080
  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    hostname: grafana
    ports:
      - 3000:3000
    depends_on:
      - prometheus

prometheus.yml

global:
  scrape_interval: 5s
  scrape_timeout: 10s
  external_labels:
    monitor: 'my-project'
rule_files:
scrape_configs:
  - job_name: myapp
    scrape_interval: 10s
    static_configs:
      - targets:
          - my_service:8080

I tried external ip as well, but i can't see my metrics in prometheus UI. Also, the target page is showing localhost:9090 is up.

What could be the problem? Can anyone correct the docker compose and prometheus file?

Thanks

prometheus docker-compose docker go
2 Answers

You're not configuring the prometheus container|service to use the prometheus.yml that you defined.

You want something like:

  prometheus:
    restart: always
    depends_on:
      - gcp-exporter
    image: prom/prometheus:latest
    container_name: prometheus
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      # Permits `curl --request POST http://localhost:9090/-/reload`
      - --web.enable-lifecycle
    volumes:
      - ${PWD}/prometheus.yml:/etc/prometheus/prometheus.yml
    expose:
      - "9090"
    ports:
      - 9090:9090

Where, in this config, the folder from which you run docker-compose (${PWD}) also contains the ${PWD}/prometheus.yml.

The YAML specifies that:

  1. the config file (using --config-file) is located in the (default?) location in the container /etc/prometheus/prometheus.yml. This line is redundant but aids documentation.
  2. the host's prometheus.yml (in ${PWD}) is mapped to the container's /etc/prometheus.prometheus.yml (matching the above config). If your folder structure differs, revise the --volume={source}/prometheus.yml:/etc/prometheus/prometheus.yml.

You can validate this by checking the Prometheus server's targets to ensure that it's scraping my_server:8080 by checking: http://localhost:9090/targets.

It would be worth ensuring that my_service is publishing metrics correctly by checking: http://localhost:8080/metrics

NOTE From your host, my_service is only accessible as localhost:8080 but, you're prometheus.yml correctly references it as my_service within the Docker Compose network.

I encourage you to not use the latest tag when referencing containers. latest is misleading. It may not reference the most recent image and it provides you very little control of which image is being used. See e.g. Understanding Docker's "latest" tag

user_6092900

So I found it. I have to set my scrape configs with the container name. something like this

scrape_configs:
  - job_name: my-service
    scrape_interval: 15s
    scrape_timeout: 10s
    metrics_path: /metrics
    static_configs:
      - targets:
          - 'prometheus:9090'
          - 'my-service:8080'

Once you fix your Prometheus volumes to your data, you will see your service is up and running at http://localhost:9090/targets

user_85024330
Related