Accessing the data interval of a Dag Run inside a task

user_8615145 0
Viewed 531

I'm building an ETL pipeline with Apache Airflow. I have to extract the latest data added to a SQL database (say daily). Therefore, I want to construct a query as follows:

SELECT foo
FROM bar
WHERE insert_date >= "DATA_INTERVAL_START_HERE"
   AND insert_date < "DATA_INTERVAL_END_HERE"

To execute this query in a task (with e.g. pyodbc), I need to access the data interval start and end time of the Dag Run object inside the extract task. How can I retrieve this information?

airflow python
1 Answers

A similar question was asked here: Airflow ETL pipeline - using schedule date in functions? However, the answer is not updated to the TaskFlow API since Airflow 2.0.

A concise way to access the data interval parameters:

@dag(schedule_interval="@daily", start_date=datetime(2022, 2, 8), catchup=True)
def tutorial_access_data_interval():
     @task()
     def extract(data_interval_start=None, data_interval_end=None, **kwargs):
          #Use data_interval_start, data_interval_end here

The Airflow engine will provide the parameters by default.

References:

https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/python.html https://airflow.apache.org/docs/apache-airflow/stable/templates-ref.html#variables

user_86151450
Related