How to prevent CloudWatch Container Insights metrics from short lived kubernetes pods started by Apache Airflow?

user_90580 0
Viewed 351

I have ~600000 metrics in my CloudWatch Metrics CWAgent namespace. 99% of them are undesired and are derived from very short lived pods.

I turns out that the amazon-cloudwatch-agent running on EKS will create a set of 5 metrics for each pod:

  • pod_memory_utilization
  • pod_number_of_container_restarts
  • pod_cpu_utilization
  • pod_network_tx_bytes
  • pod_network_rx_bytes

Since I have Apache Airflow with the KubernetesExecutor running on that EKS cluster, a large number of really short lived pod are created regularly. Each task execution gets a pod that usually last a minute or so. So I end up with 5 new metrics for each execution, usually with a single datapoint or two (because I think it's one datapoint per minute).

These metrics are totally useless for me there are too many and there are not enough data points to actually do any interesting analysis/chart.

Is there anyway of preventing these metrics to be created in the first place? I still want the metrics for my long running Pods and Deployments.

amazon-cloudwatch airflow
1 Answers

Currently the amazon-cloudwatch-agent has only one mechanism to control what gets ingested by the kubernetes plugin (plugins/processors/k8sdecorator/stores/podstore.go): the aws.amazon.com/cloudwatch-agent-ignore: true annotation in the Pod.

I don't think this annotation is officially documented but you can find it defined in the source code here and used here.

If a pod has the annotation aws.amazon.com/cloudwatch-agent-ignore: true it will be ignored by the amazon-cloudwatch-agent / cwagent and no container insights metrics will be generated for it.

You need to have amazon-cloudwatch-agent v1.247348.0 (2021-05-28) or later and your cwagent-config.json does NOT need to change, the "usual" .logs.metrics_collected.kubernetes is ok:

{
  "logs": {
    "metrics_collected": {
      "kubernetes": {
        "cluster_name": "${cluster_name}",
        "metrics_collection_interval": 60
      }
    },
    "force_flush_interval": 5
  },
  "metrics": {
    "metrics_collected": {
      "statsd": {
        "service_address": ":9125"
      }
    }
  }
}

Now in order to force Apache Airflow to add the aws.amazon.com/cloudwatch-agent-ignore: true annotation to the task/worker pods and to the pods created by the KubernetesPodOperator you will need to add the following to your helm values.yaml (assuming that you are using the "official" helm chart for airflow 2.2.3) :

airflowPodAnnotations:
  aws.amazon.com/cloudwatch-agent-ignore: "true"
airflowLocalSettings: |-
  def pod_mutation_hook(pod): 
    pod.metadata.annotations["aws.amazon.com/cloudwatch-agent-ignore"] = "true"

If you are not using the helm chart then you will need to change the pod_template_file yourself to add the annotation and you will also need to modify the airflow_local_settings.py to include the pod_mutation_hook.

user_905800
Related