Changing the auto-generated task names in dynamic tasks

Question

I'm trying to make a dynamic workflow but want to change the tasks names which airflow auto-generating it and assign to the tasks inside the list. I tried to access the context and manually change the taskid but this also not worked during the pipeline rendering in the UI.

My Code

    def get_the_route(router_ip, taskid):
        dev1 = junos_ops()
        dev1.open_fabric_connection()
        result = dev1.dev_handler.rpc.get_route_information(destination="10.0.0.3", normalize=True)

        logger.info("result is: {}".format(pformat(result)))
        dev1.close_fabric_connection()
       # <--do-some-logic-->
        return {"result": result}

    for dev in dev_list:
        get_the_route_dev_list.append(get_the_route(router_ip=dev, taskid=dev))
    start >> hello_task >> get_the_route_dev_list >> bye_task >> end

Generated Graph

Is there anyway to give different names to dynamic tasks? I know this is possible using PythonOperator. But I'm trying to do this using TaskFlow API instead.

Thanks

user_9713409 · Answer

Yes, it is possible. For example:

@task(task_id="different_name")
def foo():
    pass

with DAG(
    'test_foo',
    start_date=days_ago(1),
    schedule_interval=None,
) as dag:
    foo()

results in:

user_14203402 · Answer

You do that this way (it also works with dynamic DAGs):

@task()
def foo():
    pass

with DAG(
    'test_foo',
    start_date=days_ago(1),
    schedule_interval=None,
) as dag:
    for name in ["a", "b", "c"]:
        foo.override(task_id=name)()

You can read more here: https://airflow.apache.org/docs/apache-airflow/stable/tutorial_taskflow_api.html#reusing-a-decorated-task

Changing the auto-generated task names in dynamic tasks

2 Answers