I have deployed Airflow 2.2.2 on CeleryExecutor mode. I have built a DAG, which imports classes from a custom package. When I add a new attribute to this imported class, push a new version of my package, re-deploy Airflow with the correct new version package referenced (in a conda env), and add a reference to this new attribute in me DAG, Airflow UI shows an import error, stating that the newly added attribute does not exist in my class.
What I have already tested:
- Hard-coding the attribute value in the DAG resulted in the DAG being correctly parsed
- I verified that the correct version of my custom package was installed on the webserver/scheduler containers
- Launching python3.7 on my webserver/scheduler containers (with the conda env activated), and importing the class from my custom package resulted in no
AttributeError. - Re-deploying Airflow after my custom package deployment did not fix anything
- Deleting all entries on
dag_codeandserialized_dagmetedata tables did not fix the problem
Conclusions so far
It seems to me that this is related with the way Airflow 2.x handles dags serialization (https://airflow.apache.org/docs/apache-airflow/stable/dag-serialization.html), and that somehow the previous version of the imported class is still cached somewhere. Any help on this would be much appreciated