I am new to the observability space and I need to provide monitoring to the application on the EKS cluster which is built on fargate alone. Since Fargate does not support deamon sets, I have resolved to use Open Telemetry for collecting the metrics.
I am using the following configuration to get the various container and pod metrics from the cluster
receivers:
prometheus:
config:
global:
scrape_interval: 1m
scrape_timeout: 40s
scrape_configs:
- job_name: 'kubelets-cadvisor-metrics'
sample_limit: 10000
scheme: https
kubernetes_sd_configs:
- role: node
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_node_label_(.+)
# Only for Kubernetes ^1.7.3.
# See: https://github.com/prometheus/prometheus/issues/2916
- target_label: __address__
# Changes the address to Kube API server's default address and port
replacement: kubernetes.default.svc:443
- source_labels: [__meta_kubernetes_node_name]
regex: (.+)
target_label: __metrics_path__
# Changes the default metrics path to kubelet's proxy cadvdisor metrics endpoint
replacement: /api/v1/nodes/$${1}/proxy/metrics/cadvisor
metric_relabel_configs:
# extract readable container/pod name from id field
- action: replace
source_labels: [id]
regex: '^/machine\.slice/machine-rkt\\x2d([^\\]+)\\.+/([^/]+)\.service$'
target_label: rkt_container_name
replacement: '$${2}-$${1}'
- action: replace
source_labels: [id]
regex: '^/system\.slice/(.+)\.service$'
target_label: systemd_service_name
replacement: '$${1}'
- job_name: FTR-pod-config
scrape_interval: 5s
static_configs:
- targets: ['192.168.101.101:9090']
processors:
# add cluster name from env variable and EKS metadata
resourcedetection:
detectors: [env, eks]
batch:
timeout: 60s
exporters:
awsprometheusremotewrite:
# replace this with your endpoint
endpoint: "https://awsurl/remote_write"
# replace this with your region
aws_auth:
region: "eu-west-1"
service: "aps"
# namespace: "adot"
extensions:
health_check:
service:
pipelines:
metrics:
receivers: [prometheus]
processors: []
exporters: [awsprometheusremotewrite]
extensions: [health_check]
I need the metrics to be present per application level, With the above configuration I can fill the grafana dashboard with metrics per pod level only, i.e if I have upgraded or redeployed all these metrics go blank and I have to replace them with name of the new pod.
Is there a way to collect the metrics per application and not worry on the Pod name changing?
Update: I think we can use regex in promql to get for the complete deployment
I am collecting custom metrics like total requests, active requests and latency using the opentelemetry sdk. I am using a static config as shown in the config above.
Is there a way to streamline these as well per application instead of using static config?