We are in the process of streamlining our build/deployment pipelines for airflow codebase.
Does anyone have experience in build and deployment pipelines using CI/CD tools for apache airflow?
How, do you deploy your airflow codebase including DAGs/plugins/operators in different environments like test, staging, production etc.
How do you manage airflow.cfg config for each environments?
Where do you manage your configs for each environment.
We build all of our code into a Docker image (DAGs, plugins, different Python packages, different airflow.cfg files, etc.) that gets pushed up to our Kubernetes cluster. That same image runs everywhere, ensuring that dependencies stay locked down and each Airflow is configured best for it's use case (we run multiple Airflow instances on our Kubernetes cluster).
As far as CI/CD, since our deployment is pretty much just a docker push, we've used CircleCI without any issues.
For managing environments, we'll try to keep connections named the same across Airflows (e.g. redshift_conn) but with different credentials (dev Redshift vs prod Redshift). I think there should be more elegant solutions to that, but that's worked so far for us.
