PromQL query to show pod label in alert

user_3217396 0
Viewed 1645

I have a default alert rule in Prometheus operator as below,

   - alert: KubePodNotReady
      annotations:
        message: Pod {{`{{`}} $labels.namespace {{`}}`}}/{{`{{`}} $labels.pod {{`}}`}} has been in a non-ready state for longer than 15 minutes.
        runbook_url: https://github.com/kubernetes-monitoring/kubernetes-mixin/tree/master/runbook.md#alert-name-kubepodnotready
      expr: |-
        sum by (namespace, pod) (
          max by(namespace, pod) (
            kube_pod_status_phase{job="kube-state-metrics", namespace=~".*", phase=~"Pending|Unknown"}
          ) * on(namespace, pod) group_left(owner_kind) topk by(namespace, pod) (
            1, max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!="Job"})
          )
        ) > 0 
      for: 15m
      labels:
        severity: warning

I want the alert to show pod's label "teamname".

I am able to get the pod label with below expressions,

kube_pod_info * on(namespace, pod) group_left kube_pod_labels{label_teamname="example"} 

kube_pod_info * on(namespace, pod) group_left(label_teamname) kube_pod_labels

But I am not sure how to update the alert rule to display label.I just tried adding the label without editing expression,

  labels:
    severity: warning
    teamname: '{{ $labels.label_teamname }}'

But this didnt work.

Is changing expression required that we can include teamname in the alert? If yes,kindly suggest how to change the below expression..

  expr: |-
    sum by (namespace, pod) (
      max by(namespace, pod) (
        kube_pod_status_phase{job="kube-state-metrics", namespace=~".*", phase=~"Pending|Unknown"}
      ) * on(namespace, pod) group_left(owner_kind) topk by(namespace, pod) (
        1, max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!="Job"})
      )
    ) > 0
prometheus-operator prometheus-alertmanager promql
1 Answers

This expression works for me,

(sum by (namespace, pod) (
          max by(namespace, pod) (
            kube_pod_status_phase{job="kube-state-metrics", namespace=~".*", phase=~"Pending|Unknown"}
          ) * on(namespace, pod)   group_left(owner_kind) topk by(namespace, pod) (
            1, max by(namespace, pod, owner_kind) (kube_pod_owner{owner_kind!="Job"})
          )
        ) > 0) * on(namespace, pod) group_left(label_teamname) kube_pod_labels
user_32173960
Related