Why airflow scheduler does not run my DAG?

user_16639218 0
Viewed 602

I'm not able to run airflow DAG by scheduler. I have checked multiple threads here on forum, but I'm still not able to find the root cause. Of course DAG slider is set to ON. Below you can find DAG information:

with DAG(
        dag_id='blablabla',
        default_args=default_args,
        description='run my DAG',
        schedule_interval='45 0 * * *',
        start_date=datetime(2021, 8, 5, 0, 45),
        max_active_runs=1,
        tags=['bla']) as dag:
    t1 = BashOperator(
        task_id='blabla',
        bash_command="python3 /home/data/blabla.py",
        dag=dag
    )

I have checked cron expression which seems to be fine, start_date is hardcoded so it excludes the issue with time set to "now". When I'm checking DAGs run history all other scheduled DAGs are there listed, only this one seems to be invisible for the scheduler.

Triggering DAG manually works fine, python code works properly, there's issue only with scheduler.

What was done:

  1. checked CRON expression
  2. checked start_date whether it's hardcoded
  3. tried changing start_date to date couple months ago
  4. tried many schedule_interval values (but always daily)
  5. checked multiple threads here but did not found anything more than above bullets
airflow-scheduler airflow scheduler cron python
2 Answers

Looks okay. One thing that comes to mind is the once-a-day schedule interval, which sometimes confuses because the first run will start at the end of the interval, i.e. the next day. Since you set your start_date to more than one day ago, that shouldn't be a problem.

To find a solution, we would need more information:

  • Could you post the default_args, or your full DAG?
  • Any details about your Airflow setup (versions, executor, etc.)
  • Could you check the scheduler logs for any information/errors? Specifically, $AIRFLOW_HOME/logs/dag_processor_manager.log and $AIRFLOW_HOME/logs/scheduler/[date]/[yourdagfile.py].log
user_30664280

Issue resolved by below steps found in some other post:

try create a new python file, copy your DAG code there, rename it so that the file is unique and then test again. It could be the case that airflow scheduler got confused by the inconsistency between previous DAG Runs' metadata and the current schedule.

user_166392180
Related