Decrease Prometheus scraping interval on k8s for one Pod

user_3952994 0
Viewed 4685

Our global Prometheus scrape interval on k8s is 60s, but I want one application has 300s scrape interval.

I attach the following to my pod so the metrics are scraped.

prometheus.io/scrape: 'true'
prometheus.io/port: '{{ .Values.prometheus.port }}'
prometheus.io/path: '{{ .Values.prometheus.path }}'

Now I want to slow down the frequency of this application specifically, and tested with

prometheus.io/interval: '300s'

However it does not work. I think it requires relabel, or any other suggestion? https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus-kubernetes.yml#L251

prometheus kubernetes
2 Answers

I would like to add some details. If you use the official Prometheus Helm charts, this should be in your values-prometheus.yaml file:

# extra scraping configs
# | is required, because extraScrapeConfigs is expected to be a string
extraScrapeConfigs: |
  - job_name: 'kubernetes-service-endpoints-scrape-every-2s'

    scrape_interval: 2s

    kubernetes_sd_configs:
      - role: endpoints

    relabel_configs:
      #Custom
      - source_labels: [__meta_kubernetes_service_annotation_example_com_scrape_every_2s]
        action: keep
        regex: true

      # Boilerplate
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
        action: replace
        target_label: __scheme__
        regex: (https?)
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
      - action: labelmap
        regex: __meta_kubernetes_service_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_service_name]
        action: replace
        target_label: kubernetes_name
      - source_labels: [__meta_kubernetes_pod_node_name]
        action: replace
        target_label: kubernetes_node

To upgrade the existing(default) installation:

helm upgrade prometheus --values values-prometheus.yaml stable/prometheus

On the service or the pod you can now add these annotations:

prometheus.io/path: /metrics
prometheus.io/port: "9090"
example.com/scrape_every_2s: "true"

Remove the original prometheus.io/scrape: "true", because otherwise your service will show up as two separate Prometheus targets, which is probably not what you want.

user_3361520

In Prometheus you can set scrape_interval both globally and more specifically per job. Value defined per job overrides the global one. Compare with this example.

For more details please refer to official Prometheus documentation.

EDIT:

Below an example of a job with scrape_interval set to 300s. It selects for scraping only Pods that have example.io/should_be_scraped_every_300s: "true" annotation:

  - job_name: 'kubernetes-pods-300s'

    scrape_interval: 300s

    kubernetes_sd_configs:
    - role: pod

    # example relabel to scrape only pods that have 'example.io/should_be_scraped_every_300s: "true"' annotation
    relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_example_io_should_be_scraped_every_300s]
      action: keep
      regex: true

    # rest of this config was taken from default 'kubernetes-pods' job already present in default Prometheus config
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: true
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
    - source_labels: [__meta_kubernetes_namespace]
      action: replace
      target_label: kubernetes_namespace
    - source_labels: [__meta_kubernetes_pod_name]
      action: replace
      target_label: kubernetes_pod_name

And below fragment of Pod definition that contains appropriate annotations so that it will be discovered by Prometheus as a target to be scraped.

apiVersion: v1
kind: Pod 
metadata:
  annotations:
    example.io/should_be_scraped_every_300s: "true"
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu request for container
      buildroot'
    prometheus.io/path: /metrics
    prometheus.io/port: "9090"
    prometheus.io/scrape: "true"
...
user_117141140
Related