Configuring Prometheus in Opentelemetry on EKS with Fargate

Viewed 169

I am new to the observability space and I need to provide monitoring to the application on the EKS cluster which is built on fargate alone. Since Fargate does not support deamon sets, I have resolved to use Open Telemetry for collecting the metrics.

I am using the following configuration to get the various container and pod metrics from the cluster

receivers:
  prometheus:
    config:
      global:
        scrape_interval: 1m
        scrape_timeout: 40s

      scrape_configs:
      - job_name: 'kubelets-cadvisor-metrics'
        sample_limit: 10000
        scheme: https

        kubernetes_sd_configs:
        - role: node
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
            # Only for Kubernetes ^1.7.3.
            # See: https://github.com/prometheus/prometheus/issues/2916
          - target_label: __address__
            # Changes the address to Kube API server's default address and port
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            # Changes the default metrics path to kubelet's proxy cadvdisor metrics endpoint
            replacement: /api/v1/nodes/$${1}/proxy/metrics/cadvisor
        metric_relabel_configs:
          # extract readable container/pod name from id field
          - action: replace
            source_labels: [id]
            regex: '^/machine\.slice/machine-rkt\\x2d([^\\]+)\\.+/([^/]+)\.service$'
            target_label: rkt_container_name
            replacement: '$${2}-$${1}'
          - action: replace
            source_labels: [id]
            regex: '^/system\.slice/(.+)\.service$'
            target_label: systemd_service_name
            replacement: '$${1}'
      - job_name: FTR-pod-config
        scrape_interval: 5s
        static_configs:
          - targets: ['192.168.101.101:9090']
processors:
  # add cluster name from env variable and EKS metadata
  resourcedetection:
    detectors: [env, eks]

  batch:
    timeout: 60s
exporters:
  awsprometheusremotewrite:
    # replace this with your endpoint
    endpoint: "https://awsurl/remote_write"
    # replace this with your region
    aws_auth:
      region: "eu-west-1"
      service: "aps"
    # namespace: "adot"
extensions:
  health_check:

service:
  pipelines:
    metrics:
      receivers: [prometheus]
      processors: []
      exporters: [awsprometheusremotewrite]
  extensions: [health_check]

I need the metrics to be present per application level, With the above configuration I can fill the grafana dashboard with metrics per pod level only, i.e if I have upgraded or redeployed all these metrics go blank and I have to replace them with name of the new pod.

Is there a way to collect the metrics per application and not worry on the Pod name changing?

Update: I think we can use regex in promql to get for the complete deployment

I am collecting custom metrics like total requests, active requests and latency using the opentelemetry sdk. I am using a static config as shown in the config above.

Is there a way to streamline these as well per application instead of using static config?

0 Answers
Related