Airflow execution interval - is it standard to define the time slice using execution_date and next_execution_date?

Viewed 1022

I am new to Airflow and have been reading around to try and code my DAGs to fit the standards for the tool. Thanks to plenty of warnings I got the gist around execution_date being at the start of a time slice. Where I have been less sure is in how to handle the end of the time slice.

If I'm running a daily task to process records based on a timestamp, and especially if I want this to be idempotent, then I will need to bound the time slice at both ends. The clearest way to do this is to use execution_date and next_execution_date variables, as in the example below:

from datetime import datetime
from airflow import DAG
from airflow.providers.postgres.operators.postgres import PostgresOperator

dag = DAG(
    dag_id='time_slice_example',
    start_date=datetime(year=2021, month=2, day=1),
    schedule_interval='0 0 * * *'
)

copy_data = PostgresOperator(
    owner='airflow',
    task_id='copy_time_slice_data',
    sql='''
        INSERT INTO pipeline_tbl (id, text, other)
            SELECT id, text, other FROM daily_tbl
            WHERE data_ts >= {{ execution_date }}
            AND data_ts < {{ next_execution_date }}
    ''',
    postgres_conn_id='my_db_conn',
    dag=dag
)

(I've used a postgres query to illustrate the example but the same variables and principle would apply to any time slice operation)

So my question is whether this is normal? For all of the references to Airflow time slices, I have seen almost no examples of this approach. I can appreciate that it is arguably outside of the scope of Airflow itself, but I wanted to check that this is a standard approach and indeed that I'm not missing something more appropriate.

1 Answers
Related