How do you add scrape targets to a Prometheus server that was installed with Kubernetes-Helm?

user_6905609 0
Viewed 8578

Background

I have installed Prometheus on my Kubernetes cluster (hosted on Google Container Engineer) using the Helm chart for Prometheus.

The Problem

I cannot figure out how to add scrape targets to the Prometheus server. The prometheus.io site describes how I can mount a prometheus.yml file (which contains a list of scrape targets) to a Prometheus Docker container -- I have done this locally and it works. However, I don't know how to specify scrape targets for a Prometheus setup installed via Kubernetes-Helm. Do I need to add a volume to the Prometheus server pod that contains the scrape targets, and therefore update the YAML files generated by Helm??

I am also not clear on how to expose metrics in a Kubernetes Pod -- do I need to forward a particular port?

prometheus google-kubernetes-engine kubernetes-helm kubernetes
2 Answers

First of all, you need to create a Service Monitor, which is a custom K8s resource. Just create a servicemonitor.yaml in the manifests folder.

Since when we are deploying on K8s, we don't have access to the Prometheus.yaml file to mention the targets, we create the servicemonitor, which in-turn adds the target to the scrap_config in the Prometheus.yaml file. You can read about it more from here.

This is a sample servicemonitor.yaml file for exposing Flask App metrics in Prometheus.

apiVersion: monitoring.coreos.com/v1 
kind: ServiceMonitor 
metadata:
  name: flask-metrics
  namespace: prometheus # namespace where prometheus is running
  labels:
    app: flask-app
    release: prom  # name of the release 
    # ( VERY IMPORTANT: You need to know the correct release name by viewing 
    # the servicemonitor of Prometheus itself: Without the correct name, 
    #  Prometheus cannot identify the metrics of the Flask app as the target.)
spec:
  selector:
    matchLabels:
      # Target app service
      app: flask-app # same as above
      release: prom # same as above
  endpoints:
  - interval: 15s # scrape interval
    path: /metrics # path to scrape
    port: http # named port in target app
  namespaceSelector:
    matchNames:
    - flask # namespace where the app is running

Also add this Release Label to the Services and Deployments file too, in the metadata and spec section.

If you encounter a situation where Prometheus is showing the Target but not the endpoints, take a look at this: https://github.com/prometheus-operator/prometheus-operator/issues/3053

Some useful links:

user_113234730
Related