Disable Airflow mini-scheduler / fast-follow

user_1011253 0
Viewed 276

I'm running Airflow v2.2.3 on Kubernetes.

I have a dedicated scheduler pod, a dedicated web-server pod and I'm using Kubernetes Executors.

I have ~4,000 DAGs, created dynamically in a for-loop.

I read Astronomer's post about "fast-follow" (also called "mini-scheduler") and I wonder if there's an option to disable it via configuration or something.

The thing is, that it comes with a penalty where:

  • Dynamic DAGs are being calculated per each task operator, as a new fresh executor is running every time - I never saw reuse of Kubernetes executor (and if it using pod's patch verb for that - I guess that the dynamic DAGs would recalculated again anyway).
  • The time it takes to recalculate dynamic DAGs lead to a "queued" status for every task - which mean that a lot of time is spent just for recalculating the DAGs over and over again.
  1. Is there a configuration (or any other way) to disable the mini-scheduler?
  2. Am I using the KubernetesExecutor wrong? should it stay up for more than one task (and thus calculate the dags only at the beginning)
  3. I'm running a multi-tenant setup, where each one of my customers (businesses) run it's own DAG - but they all look the same (except input arguments + schedule time). Should I take a different approach? (other than using a dedicated DAG per customer).

Thanks

airflow-2.x airflow-scheduler airflow
1 Answers

Answering your questions:

1. This feature introduced in PR To disable the feature set in airflow.cfg:

[scheduler]
schedule_after_task_execution = False

  1. Disabling the feature may not be advised as explained in in config.yml description. It may effect performance.

  2. Airflow doesn't support multi tenant setup yet. It's currently in discussions under AIP 1 (with sub AIPs 33-34). That said - I'm not really sure how this question relates to the issue you described before. It's OK to use dedicated DAG per customer. It even makes sense as one customer can backfill / clear previous tasks and you don't want it to cause problems for other customers. This is more a design decision. I don't think there is a right or wrong here - it depends how you organization decide to work.

user_146244090
Related