I would like to be able to unit test my airflow operators without having to run airflow db init before the tests. Is there a way to do this ?
Thank you very much for your help !
2 Answers
Not a direct answer but a different approach on this.
We have the same issue and still experimenting on this. For now we have decided to split the functionality of the custom airflow operator in two parts. Operator itself handling anything that has to do with the db and orchestrates the process and in another module we have the logic of the operator - that has nothing to do with the database.
For example, assume that you have an operator that gets a csv file from a bucket performs transformations and then inserts the file in the database. Lets call it CsvToDBOperator. For this we would have two files/classes.
- CsvToDBOperator
- CsvToDBUtils
The second one will keep all the methods/functions that are responsible for the transformations. The first one will be responsible to get the file from the bucket, pass it to the second for the transformation and then getting the result from it and finally inserting it in the DB.
Like this you can write unit tests for CsvToDBUtils class and only create unit tests on those.
Airflow is already testing majority of operators without having to init db. We are heavily use mocking:
It common to split hook and operator testing. So, first you test hook methods (example DataprocHook method test) then you test the operators using the hook by mocking it (corresponding Dataproc example).
Related