Airflow DAG not triggered at schedule time

user_12491814 0
Viewed 1091

Below is my DAG code, it is not getting triggered @scheduled time. If i run manually, it is working fine. Not sure where is the problem. I tried testing two to three corn expression but without any luck.

from airflow import DAG
from airflow.operators.bash_operator import BashOperator
#from datetime import timedelta

YESTERDAY = datetime.datetime.now() - datetime.timedelta(days=1)

default_args = {
    'start_date': airflow.utils.dates.days_ago(0),
    'retries': 1,
    'retry_delay': datetime.timedelta(minutes=5)
    }

dag = DAG(
    'airflow_worker_pod_monitoring',
    default_args=default_args,
    description='liveness monitoring dag',
    schedule_interval='0,00 15 * * *',
    start_date=YESTERDAY,
    dagrun_timeout=datetime.timedelta(minutes=60))

# priority_weight has type int in Airflow DB, uses the maximum.
t1 = BashOperator(
    task_id='monitor_pod',
    bash_command='bash /home/airflow/gcs/data/testscript.sh' ,
    dag=dag,
    depends_on_past=False,
    priority_weight=2**31-1)```

How to make this working?
google-cloud-composer airflow-scheduler google-cloud-platform
2 Answers

There are 4 potential causes of this:

  1. schedule_interval provided in default_args
  2. Existing DAG's schedule_interval was modified
  3. start_date is set to datetime.now()
  4. start_date not aligned with schedule_interval

Your issue could be caused by 2. or 4. above since 1. & 3. are properly configured.

user_118661330

Airflow documentation recommends using static data https://airflow.apache.org/docs/apache-airflow/stable/faq.html#what-s-the-deal-with-start-date . Using dynamic date (such as airflow.utils.dates.days_ago(0)) as start_date is not advisable and may cause issues as the dag gets confused at 00:00 and switch to next day incorrectly. Set your start_date to be a fixed datetime and it should work correctly.

In general, the way the Airflow scheduling works is not always straight forward. I recommend this article going deeper into this topic https://towardsdatascience.com/airflow-schedule-interval-101-bbdda31cc463

user_152603950
Related