Assuming I have DAG with dynamic tasks. I generated tasks based on schedule time. My Dag run every hour, so every run total of tasks are different. exec_date=2021-04-01 00:00:00 generate 1 task with name task_1 exec_date=2021-04-01 01:00:00 generate 2 task with name task_1, task_2 etc.
Is it okay to do that?
This is not a good practice. The docs state:
Workflows are expected to be mostly static or slowly changing. You can think of the structure of the tasks in your workflow as slightly more dynamic than a database structure would be. Airflow workflows are expected to look similar from a run to the next, this allows for clarity around unit of work and continuity.
You can create dynamic tasks but not when the DAG structure is changed every run. What you can do is generate all tasks in a static manner and during execution add Sensor/Branch to check if the task should run or being skipped.
