How to read airflow variables into kubernetes pod instance

user_9426482 0
Viewed 494

I am trying to read airflow variables into my ETL job to populate variables in the curation script. I am using the KubernetesPodOperator. How do I access the metadata database from my k8's pod?

Error I am getting from airflow: ERROR - Unable to retrieve variable from secrets backend (MetastoreBackend). Checking subsequent secrets backend.

This is what I have in main.py for outputting into the console log. I have a key in airflow variables named "AIRFLOW_URL".

  from airflow.models import Variable
    AIRFLOW_VAR_AIRFLOW_URL = Variable.get("AIRFLOW_URL")

    logger.info("AIRFLOW_VAR_AIRFLOW_URL: %s", AIRFLOW_VAR_AIRFLOW_URL)

Can anyone point me in the right direction?

airflow kubernetes
2 Answers

Your DAG can pass them as environment variables to your Pod, using a template (e.g. KubernetesPodOperator(... env_vars={"MY_VAR": "{{var.value.my_var}}"}, ...)).

user_35400740

It looks like you have a secrets backend set in config without having a secrets backend set up, so Airflow is trying to go there to fetch your variable. See this link.

Alter your config to remove the backend and backend_kwargs keys, and it should look at your Airflow variables first.

[secrets]
backend =
backend_kwargs =
user_182997500
Related