Different way to set Airflow DAG fields

user_7876879 0
Viewed 51

I would like to create an Airflow DAG and want to learn which parameters should be set in field_1 vs default_args vs args?

my_dag = DAG(
  "my_dag",
  "field_1"="xxx",
  default_agrs=default_args,
  **args
)

I checked with documentation, I understand that some parameters such as "owner" have to be set through the default_args and can't be in field_1. But looks like there's no difference for the most of parameters. I tested some fields such as "catchup" and "on_failure_callback", and they all work in all these three places.

So I wonder what's the best practice of setting parameters when create a dag?

airflow
1 Answers

The best practice is something like the Airflow tutorial

with DAG(
    'tutorial',
    # These args will get passed on to each operator
    # You can override them on a per-task basis during operator initialization
    default_args={
        'depends_on_past': False,
        'email': ['airflow@example.com'],
        'email_on_failure': False,
        'email_on_retry': False,
        'retries': 1,
        'retry_delay': timedelta(minutes=5),
        # 'queue': 'bash_queue',
        # 'pool': 'backfill',
        # 'priority_weight': 10,
        # 'end_date': datetime(2016, 1, 1),
        # 'wait_for_downstream': False,
        # 'sla': timedelta(hours=2),
        # 'execution_timeout': timedelta(seconds=300),
        # 'on_failure_callback': some_function,
        # 'on_success_callback': some_other_function,
        # 'on_retry_callback': another_function,
        # 'sla_miss_callback': yet_another_function,
        # 'trigger_rule': 'all_success'
    },
    description='A simple tutorial DAG',
    schedule_interval=timedelta(days=1),
    start_date=datetime(2021, 1, 1),
    catchup=False,
    tags=['example'],
) as dag:
...

Reference: https://airflow.apache.org/docs/apache-airflow/stable/tutorial.html#example-pipeline-definition

But I use something like that and is enough:

import pendulum

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=1)
}

with DAG(
    default_args=default_args,
    dag_id='dag_etl',
    catchup=False,
    start_date=pendulum.datetime(year=2022, month=1, day=1, tz='America/Chicago'),
    schedule_interval='0 8 * * *',  # https://crontab.guru/#0_8_*_*_*
    description='DAG Extract Transform Load'
) as dag:
...
user_108848220
Related