Backfill with --mark_success is slow

Question

I expect the behaviour of backfill with -m to run more or less instantaneous, as no work is done. However I am experiencing that the scheduler spends about 30 seconds on each task, which seems excessive.

I am running a small airflow instance in local_executor mode with Postgresql as DB.

Does this reflect your experience with Airflow backfill? Do you know about performance pitfalls or tricks to speed it up?

user_5152989 · Answer

I have found one speed-up:

--donot-pickle
Do not attempt to pickle the DAG object to send over to the workers, just tell the workers to run their version of the code

Default: False

https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#Named%20Arguments_repeat8

Backfill with --mark_success is slow

1 Answers