We are planning to switch from managing airflow ourselves to Managed Apache Airflow services of AWS. Our original dags use some custom environment variables that need to be set in Managed airflow as well. So far I was not able to find a way to set custom environment variables while setting up airflow environment in MWAA. Please let me know if anyone knows how to set them.
3 Answers
An approach for setting environment variables is to use Airflow Variables.
In MWAA, you can store Airflow Variables in AWS Secrets Manager. This approach is documented in MWAA's official documentation. Note that this approach requires specific configuration for the MWAA environment. Once the MWAA environment is set up and the variables are stored in AWS Secrets Manager, the variables become accessible through the Airflow Variable APIs.
from airflow.models import Variable
# Normal call style
foo = Variable.get("foo")
See Step two: Create the Secrets Manager backend as an Apache Airflow configuration option
See Step four: Add the variables in Secrets Manager
you should be able to set core.myconfig env variable. mwaa accepts it even though its not on the list. mwaa will create AIRFLOW__CORE__MYCONFIG env variable.
You could also create a custom plugin that generates runtime environment variables.
Link Ref: https://docs.aws.amazon.com/mwaa/latest/userguide/samples-env-variables.html
Related