How to access the credentials across DAG tasks in airflow without using connections/variables

user_13017197 0
Viewed 695

Consider I am having multiple DAG in Airflow.

Every task in the DAG tries to execute presto queries, I just override the get_conn() method in the airflow. On each call of the get_conn() method, it gets credentials from the AWS secrets manager.

The maximum request to the secrets manager is 5000. In this case, I need to cache my credentials somewhere(Should not use Connections/Variables, DB, S3), so that they can be used across all tasks without calling the secrets manager.

My question here is,

Is there any way we can handle those credentials in our code with Python/Airflow by calling get_conn() at once?

network-storage aws-secrets-manager airflow caching python-3.x
1 Answers

You could write your own custom secret backend https://airflow.apache.org/docs/apache-airflow/stable/security/secrets/secrets-backend/index.html#roll-your-own-secrets-backend extending the AWS one and overriding the methods to read the credentials and store it somewhere (for example in local file or a DB as caching mechanism).

If you are using local filesystem however, you have to be aware that your caching reuse/efficiency will depends on how your tasks are run. If you are running a CeleryExecutor, then such local file will be available for all processes running on the same worker (but not to celery processes running on other workers). If you are running KubernetesExecutor, each task runs in it's own Pod, so you'd have to mount/map some persistent or temporary storage to inside your PODS to reuse it. Plus you have to somehow solve the problem of concurrent processes writing there and refreshing such cache periodically or when it changes.

Also you have to be extra careful as it brings some issues regarding the security as such local cache will be available to all DAGs and python code run in tasks even if they are not using the connection (so for example Airflow 2.1+ built-in automated secret masking will not work in this case and you have to be careful not to print the credentials to logs.

user_5167010
Related