Apache Airflow - Dag doesn't start even with start_date and schedule_interval defined

user_6534180 0
Viewed 829

I am new at Airflow but I've defined a Dag to send a basic email every day at 9am. My DAG is the following one:

from airflow import DAG
from datetime import datetime, timedelta
from airflow.operators.bash_operator import BashOperator
from airflow.operators.email_operator import EmailOperator
from airflow.utils.dates import days_ago

date_log = str(datetime.today())
my_email = ''
default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
    'start_date': days_ago(0),
    'email': ['my_email'],
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 1,
    'retry_delay': timedelta(minutes=5),
    'concurrency': 1,
    'max_active_runs': 1
}

with DAG('TEST', default_args=default_args, schedule_interval='0 9 * * *',max_active_runs=1, catchup=False) as dag:
    t_teste = EmailOperator(dag=dag, task_id='successful_notification',
        to='my_email',
        subject='Airflow Dag ' + date_log,
        html_content="""""")
    t_teste

I've all the configurations as I needed, and I have the webserver and scheduler running. Also, I have my Dag active on UI. My problem is that my DAG seems to be doing nothing. It hasn't run for two days, and even if it passes the scheduled time, it doesn't run as expected. I have already tested and run my trigger manually, and it ran successfully. But if I wait for the trigger time, it does nothing.

Do you know how what I am doing wrong?

Thanks!

airflow-scheduler airflow
1 Answers

Your DAG will never be scheduled. Airflow schedule calculates state_date + schedule_interval and schedule the DAG at the END of the interval.

>>> import airflow
>>> from airflow.utils.dates import days_ago
>>> print(days_ago(0))
2021-06-26 00:00:00+00:00

Calculating 2021-06-26 (today) + schedule_interval it means that the DAG will run on 2021-06-27 09:00 however when we reach 2021-06-27 the calculation will produce 2021-06-28 09:00 and so on resulting in DAG never actually runs.

The conclusion is: never use dynamic values in start_date!

To solve your issue simply change:

'start_date': days_ago(0) to some static value like: 'start_date': datetime(2021,6,25) note that if you are running older versions of Airflow you might also need to change the dag_id.

user_146244090
Related