How to version control Airflow variables and connections?

user_9583433 0
Viewed 594

From a development perspective, defining variables and connections inside the UI is effective but not robust, as it is impossible to keep track of what has been added and removed.

Airflow came up with a way to store variables as environment variables. But a few natural questions arise from this:

  1. Does this need to be defined before every DAG? What if I have multiples DAGs sharing the same env values? Seems a bit redundant to be defining it every time.
  2. If defined this way, do they still display on the UI? The UI is still a great idea for taking quick look at some of the key value pairs.

I guess in a perfect world, the solution I would be looking for is somehow, just define the value of the variables and connections in the airflow.cfg file which would automatically populate the variables and connections in the UI.

Any kind of help is appreciated. Thank you in advance!

airflow-scheduler airflow
1 Answers

There is one more way of storing and managing and connections, one that is most versatile, secure and gives you all the versioning and auditing support - namely Secret Backends.

https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html

It has built-in integration with Vault, GCP Secret Store, AWS Secret store, you can use Local Filesystem Secret Backend, and you can also roll your own backend.

When you use one of those then you get all the versioning, management, security, access management coming from the Secret Backend you use (most of the secret backends have all those built-in).

This also means that you CANNOT see/edit the values via Airflow UI as it's all delegated to those backends. But the backends usually come with their own UIs for that.

Answering your questions:

  1. If you define connections/variables via env vars, you should define the variables in your Workers and Scheduler, not in the DAGs. That means that (if your system is distributed) you need to have a mechanism to update those variables and restart all airflow processes when they change (for example via deploying new images with those variables or upgrading helm chart or similar)

  2. No. The UI only displays variables/connections defined in the DB.

user_5167010
Related