Scape metrics from multiple containers in prometheus with Istio

Question

Our application is deployed in the istio service mesh and we are trying to scrape metrics at container level using the prometheus.io annotations.

So we have enabled spring boot metrics in our application and we are able to fetch the metrics on the given path '/manage/prometheus'. We have enabled Prometheus annotations in the deployment file of our application as follows:

   metadata:
      annotations:
        prometheus.io/scrape: 'true'
        prometheus.io/port: '8080'
        prometheus.io/path: '/manage/prometheus'

This works fine when there is a single container in the pod. But for pods that have multiple containers, we are unable to scrape the metrics with the container port. Following are the workarounds we tried:

Following the reference https://gist.github.com/bakins/5bf7d4e719f36c1c555d81134d8887eb we tried to add the relabel configs for scraping data at container level:

prometheus-config.yaml

   scrape-configs:
    - job_name: kubernetes-pods
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - action: keep
        regex: true
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scrape
      - source_labels: [__meta_kubernetes_pod_container_port_name]
        action: keep
        regex: (.*)
      - source_labels: [ __address__, __meta_kubernetes_pod_container_port_number]
        action: replace
        regex: (.+):(?:\d+);(\d+)
        replacement: ${1}:${2}
        target_label: __address__
      - action: replace
        regex: (https?)
        source_labels:
        - __meta_kubernetes_pod_annotation_prometheus_io_scheme
        target_label: __scheme__
        - action: replace
          regex: (.+)
          source_labels:
          - __meta_kubernetes_pod_annotation_prometheus_io_path
          target_label: __metrics_path__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - action: replace
        source_labels:
        - __meta_kubernetes_namespace
        target_label: kubernetes_namespace
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_name
        target_label: kubernetes_pod_name
      - action: drop
        regex: Pending|Succeeded|Failed
        source_labels:
        - __meta_kubernetes_pod_phase
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_name
        target_label: container
      - action: replace
        source_labels:
        - __meta_kubernetes_pod_container_port_number
        target_label: container_port

But after applying the above configuration we are getting the error as:

Get "http://10.x.x.x:8080/stats/prometheus": read tcp 10.y.y.y:45542->10.x.x.x:8080: read: connection reset by peer

So 10.x.x.x is the pod IP and 8080 is the container port, it is not able to scrape using the container port.

We tried the above configuration after removing the istio mesh i.e. by removing the istio sidecar from all the microservices pods and we could see container level metrics being scraped.

Istio’s proxy is somewhere blocking the metrics to be scraped at the container level.

Have anyone faced this similar issue?

Scape metrics from multiple containers in prometheus with Istio

0 Answers