How to handle errors for AIRFLOW (DatabricksSubmitRunOperator)

user_16034298 0
Viewed 392

I have Airflow dag , which runs series of Databricks notebook.

Now What i want is , if any one the notebook got failed? How to send mail to user that this notebook got failed with few details like execution date.

and is there any error handling way?

airflow azure-databricks
2 Answers

The BaseOperator has parameters which allow to configure sending emails in case of failure thus it's available to all operators:

DatabricksSubmitRunOperator(...,email_on_failure=True,
                            email='your.email@where.com')
user_146244090

Step1: Set email_on_failure to False and use the operators’s on_failure_callback. on_failure_callback the function described below.

from airflow.utils.email import send_email

def notify_email(contextDict, **kwargs):
    """Send custom email alerts."""

    # email title.
    title = "Airflow alert: {task_name} Failed".format(**contextDict)

    # email contents
    body = """
    Hi Everyone, <br>
    <br>
    There's been an error in the {task_name} job.<br>
    <br>
    Forever yours,<br>
    Airflow bot <br>
    """.format(**contextDict)

    send_email('you_email@address.com', title, body)

Step2: Short example airflow dag below

from airflow.models import DAG
from airflow.operators import PythonOperator
from airflow.utils.dates import days_ago

args = {
  'owner': 'me',
  'description': 'my_example',
  'start_date': days_ago(1)
}

# run every day at 12:05 UTC
dag = DAG(dag_id='example_dag', default_args=args, schedule_interval='0 5 * * *')

def print_hello():
  return 'hello!'

py_task = PythonOperator(task_id='example',
                         python_callable=print_hello,
                         on_failure_callback=notify_email,
                         dag=dag)

py_task

Note where set on_failure_callback equal to notify_email in the PythonOperator.

user_111048050
Related