The concepts of Apache Airflow state that there should not be any expensive database calls in the top-level code of DAGs, because .py files are parsed every min_file_process_interval seconds. On the other hand, I read that your DAGs (or workflow) should not change that much, so I do not understand why files are supposed to be parsed that often? (default is 30 seconds)
Our use case is that we would like to control the schedule interval of DAGs via Variables in Airflow's UI. Our DAGs hardly change and we do not add new DAGs that often. Currently, we are looking to create around 30-50 DAGs and each of them should contain this Variable.get() call to get the presently set interval time. We are okay with setting min_file_process_interval to some high number e.g. twice or once a day. What are the consequences of this?
We might change the intervals of DAGs once a month and we would like to not edit the .py files. Are there other recommended ways to set intervals without editing the code?