Clear tasks in Airflow

Viewed 472

Initial question - https://stackoverflow.com/questions/70623990/retraining-a-machine-learning-model-using-airflow-pipeline

I have tried to clear a particular task and its downstream using the bash operator. However what I see is that, after clear task, this particular task is rerun in all the previous DAG runs. What I require is the task to rerun in the current DAG run only , not in the previous ones.

I tried to do this using the airflow UI, there the behaviour is as expected !

from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta

# Default settings applied to all tasks
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'email_on_failure': False,
    'email_on_retry': False,
}

with DAG('clear_upstream_task',
         start_date=datetime(2021, 1, 1),
         schedule_interval=None,
         default_args=default_args,
         catchup=False,
         ) as dag:
    t0 = DummyOperator(
        task_id='t0'
    )
    t1 = DummyOperator(
        task_id='t1'
    )
    t2 = DummyOperator(
        task_id='t2'
    )
    t3 = BashOperator(
        task_id='t3',
        bash_command='airflow tasks clear -t t1 -d -y clear_upstream_task'
    )

    t0 >> t1 >> t2 >> t3
0 Answers
Related