In Airflow, how do we pass environment variables to Kubernetes Executor worker pods?

user_4677318 0
Viewed 390

I am using Airflow 1.0.15. It is deployed on a Kubernetes cluster using the Airflow official helm charts. And i'm not able to pass an environment variable to the worker pods.

The extraEnv added in the helm chart were added to the scheduler and web pods but not the worker pods which use Kubernetes Executor.

I was able to mount a configmap into the worker pod using the executor_config parameter. But how do we pass an environment variable to a worker pod?

airflow-scheduler airflow kubernetes environment-variables
1 Answers

I hit the same problem trying to pass environment variables to the KubernetesExecutor pods using this airflow-helm community chart.

After a few dead ends trying to set variables via config map and as per airflow variables doc and airflow config doc e.g.:

airflow:
  config:
    AIRFLOW__SCHEDULER__DAG_DIR_LIST_INTERVAL: "30"
    my_env_var: "hello this is your env var"
    AIRFLOW__my_env_var: "hello this is your env var"
    AIRFLOW__VAR__my_env_var: "hello this is your env var"

I found that setting environment varibales within the below extraEnv: block in helm values.yaml works following the documentation in the faq. Setting extaEnv results in values mapping from helper pods.tpl into the k8 pod template file.

airflow:
  extraEnv:
    - name: my_env_var
      value: "hello this is your env var"
    - name: my_other_env_var
      valueFrom:
        secretKeyRef:
          name: my-kubectl-secret
          key: value

In my case, I create secrets with kubectl or substitute in the values as part of the deployment pipeline. The downside is that my helm chart needs to be updated for each new secret / environment variables.

Lastly, these are not set as Airflow variables. They are standard environment variables on kubernetes pod os. In your DAGs and tasks you access them e.g.

def task():
    import os
    var = os.getenv('my_env_var')
    return {
        'message': var
    }

This SO answer suggests another method using config-maps.

user_45380660
Related