Airflow task in up_for_reschedule status but does not release worker

Viewed 290

I have a DAG with an HttpSensor set to mode="reschedule". This task is hitting an API that will return True after 100 seconds for a specified run. I thought that setting my sensor to mode="reschedule" would free the worker so that it could pick up a new task or start working on a new run. This does not seem to be the case, as I can have 16 parallel Runs whether they are in up_for_reschedule or running status.

Am I misunderstanding what mode="reschedule" does?

Or is there a limit to the number of concurrent runs I can have set elsewhere? If so, is there a way to set the number of workers (so as to not run out of resources) but have an unlimited number of runs?

The rescheduled task:

    wait_for_sleep = HttpSensor(
        task_id='http_sensor_check',
        http_conn_id='',
        endpoint="http://0.0.0.0:8081/long_sleep/{{run_id}}",
        request_params={},
        response_check=lambda response: response.text == "True",
        poke_interval=60,
        dag=dag,
        mode='reschedule',
    )

16 up_for_reschedule tasks

16 running tasks

I also scaled up the number of concurrent runs a DAG could have, so that it exceeded the number of max_active_tasks_per_dag. Thinking this would mean 16 tasks in the running state and an unlimited number in the up_for_reschedule state. However, it looks like that is not the case.

number of active tasks blocked by up_for_reschedule tasks

0 Answers
Related