hello world of Apache Airflow not working

user_6707010 0
Viewed 611

I am learning Apache Airflow and trying to write hello world version of it.

I have setup up apache airflow version 2.0 with python version 3.8and below is code containing dag and task

import datetime
import logging

from airflow import DAG
from airflow.operators.python import PythonOperator


def greet_hello():
    logging.info("Hello World")


dag = DAG("FirstDag", start_date=datetime.datetime.now(),schedule_interval=None)

first_task = PythonOperator(python_callable=greet_hello , dag=dag , task_id="first-task")

I have configured apache airflow in the following way

  1. pip install apache-airflow in venv
  2. airflow create user ....
  3. airflow db init
  4. airflow webserver
  5. airflow scheduler

I can see the Dag in the UI

enter image description here

But when I check the task instance details I see below message

"Task Instance State Task is in the 'None' state which is not a valid state for execution. The task must be cleared in order to be run."

Even though the scheduler is running I am getting below message in UI

enter image description here

When I go to graph view , and hover over first-task , it says not yet started. I am struggling to figure out what wrong I am going and how to resolve it. Thank you

airflow
2 Answers

I believe there are two issues here.

  1. The DAG itself appears to be paused. You'll need to set it to Active in order for the task to be scheduled.

  2. The scheduler alert is likely giving you slightly misleading information. Airflow involves multiple separate processes running concurrently. It uses a "heartbeat" to keep track of whether the scheduler process is alive. In other words, the scheduler is checking in with another airflow process on a periodic interval. If the scheduler hasn't made a heartbeat within a predetermined amount of time, Airflow isn't sure if the scheduler is still alive it puts up this error.

The reason this is misleading is that it could just mean that the scheduler is configured to heartbeat less frequently than airflow the "healthy" cutoff time interval. This often happens when using airflow locally. The default configuration is set up to minimize system resources, so the heartbeat can be infrequent.

This is straightforward to check.

  1. Check to see what value airflow is using to determine if the scheduler has made a heartbeat recently enough.
$ airflow config get-value scheduler scheduler_health_check_threshold

30

On my machine, airflow considers the scheduler to be healthy if it has returned a heartbeat in the last 30 seconds.

  1. Check the heart rate value. This is how often the scheduler checks in to, among other things, indicate that it's healthy.
$ airflow config get-value scheduler scheduler_heartbeat_sec

5

Here, airflow checks in every 5 seconds.

If scheduler_heartbeat_sec > scheduler_health_check_threshold, you'll see the message you indicated. That doesn't mean that anything is broken, it just means that airflow is unsure whether the scheduler is alive.

  1. To make sure the scheduler is running, check whatever terminal window you're using to run the scheduler. If it's outputting logs on regular intervals and you don't see any errors, it should be working properly.

  2. To check the last time the scheduler made a heartbeat http://localhost:8080/health (or whatever port you're using locally). Don't worry about the status value, since that's generated with the same logic surrounding heartbeat and heartrate. But latest_scheduler_heartbeat will tell you when the scheduler most recently had a heartbeat.

user_21417680
user_56915250
Related