Initial question - https://stackoverflow.com/questions/70623990/retraining-a-machine-learning-model-using-airflow-pipeline
I have tried to clear a particular task and its downstream using the bash operator. However what I see is that, after clear task, this particular task is rerun in all the previous DAG runs. What I require is the task to rerun in the current DAG run only , not in the previous ones.
I tried to do this using the airflow UI, there the behaviour is as expected !
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from airflow.operators.bash_operator import BashOperator
from datetime import datetime, timedelta
# Default settings applied to all tasks
default_args = {
'owner': 'airflow',
'depends_on_past': False,
'email_on_failure': False,
'email_on_retry': False,
}
with DAG('clear_upstream_task',
start_date=datetime(2021, 1, 1),
schedule_interval=None,
default_args=default_args,
catchup=False,
) as dag:
t0 = DummyOperator(
task_id='t0'
)
t1 = DummyOperator(
task_id='t1'
)
t2 = DummyOperator(
task_id='t2'
)
t3 = BashOperator(
task_id='t3',
bash_command='airflow tasks clear -t t1 -d -y clear_upstream_task'
)
t0 >> t1 >> t2 >> t3