prometheus past metrics not shown on target node restart

user_7655242 0
Viewed 470

I am new to Prometheus and need help to understand why past metric data is not shown when the target node restarts.

I have set up a Golang web server (target). This server makes use of the Go Prometheus Docs Golang Prometheus client to prepare metrics and exposes metrics on port 3000. Prometheus scrapes data from this target.

Prometheus Config file:

global:   scrape_interval: 10s   scrape_timeout: 10s
    scrape_configs:
  - job_name: 'webServer1'
    static_configs:
    - targets: ['webServer1:8080']

I have Also set the retention flag in docker-compose

prometheus:
image: prom/prometheus
volumes:
  - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
ports:
  - "127.0.0.1:9090:9090"
command:
  - '--config.file=/etc/prometheus/prometheus.yml'
  - '--storage.tsdb.path=/prometheus'
  - '--web.console.libraries=/etc/prometheus/console_libraries'
  - '--web.console.templates=/etc/prometheus/consoles'
  - '--storage.tsdb.retention.time=200h'
  - '--web.enable-lifecycle'

I have instrumented a web server (target) to count the number of HTTP requests made to /bar endpoint. I can see the correct request count on Prometheus (click on image 1 link).

image 1

But on webserver restart, previously recorded metrics are not shown on Prometheus (click on image 2 link).

image 2

It's unclear to me why metrics earlier scraped from the webserver (target) are not shown above on target node restart. I can see previously scraped metrics in graph view (see image 3 link). But not in the table view.

image 3

prometheus go monitoring
1 Answers

Looks like you made the hostname part of the metric name. This produces new metrics for every container. The table view only shows metrics that were contained in the most recent scrape for each target.

To fix the issue remove the hostname part from the metric name so the names don't change between restarts. If this is really useful information, add them as a label instead, although that is almost certainly a bad idea.

user_6033160
Related