Node selector (affinities/tolerations) for Airflow base container

Question

I select k8s node for KubernetesPodOperator from values.yaml file and from DAG. It works well, but I would like to use different nodes per DAG but changing tolerations/affinities in DAG file would run base container based on settings from values.yaml file. Any idea how to control where base container runs from DAG file?

Thanks!

mrp1nk · Answer

When using KubernetesPodOperator based task you can define affinity for that task:

https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py#L116 - where the arg should be defined
https://airflow.apache.org/docs/apache-airflow-providers-cncf-kubernetes/stable/_modules/tests/system/providers/cncf/kubernetes/example_kubernetes.html#affinity - how to prepare your own affinity config that needs to be passed to affinity arg in KubernetesPodOperator (this file provides complete DAG example with task configured with custom affinity)

Node selector (affinities/tolerations) for Airflow base container

1 Answers