Value of Airflow variable gets invalid on restarting docker container

user_12837220 0
Viewed 1439

There is a long list of dags and associated airflow variables at some remote instance of airflow, copy of which is running in my local system. All the variables from remote airflow instance are imported to my local airflow instance.

I have installed airflow image on top of docker and thereafter started the container. Everything works fine and I can access the airflow UI from my local system.

Problem: Whenever I restart the airflow container, all the variables that were imported during the previous container run get invalid like this. enter image description here

Work Around Import the variables again to fix the variable related error.

However, Its really frustrating to import variables every time container starts. There must be an intelligent way of achieving this. Please help me understand what am I doing wrong.

airflow docker python
2 Answers

The Root Cause of this Problem is the AirFlow Encryption Mechanism for the Key-value Variables. when you import your variables manually, their is_encrypted attributes, are automatically set to True.

Whenever, you restart the container, new Encryption Key is generated, thus the old ones got Invalid.

You Have 3 Options :

I personally chose the second one, so my docker-compose.yml file, looks like this :

       environment:
           - LOAD_EX=n
           - EXECUTOR=Local
           - AIRFLOW__CORE__FERNET_KEY='81HqDtbqAywKSOumSha3BhWNOdQ26slT6K0YaZeZyPs='

thanks to wittfabian

user_17437240

New encryption key is generated when the docker container is restarted.

To ensure that the same encryption key is used you will have to either hardcode a FERNET_KEY in the config file or pass an env variable when the container is initially run.

docker run -it -p 8888:8080 -v D:\dev\Dags:/usr/local/airflow/dags -e FERNET_KEY=81HqDtbqAywKSOumSha3BhWNOdQ26slT6K0YaZeZyPs= --name my_airflow_dags airflow_image

The Fernet key here can be anything. Once this key is provided, docker can reuse the same every time container is restarted.

user_65635670
Related