How to monitor Spark job with Airflow

user_166836 0
Viewed 4577

I set up a few dags, which eventually ends with a spark-submit command to a spark cluster. I'm using cluster mode if that makes a difference. Anyways, so my code works, but I realized if the spark job were to fail, I wouldn't necessarily know from within the Airflow UI. By triggering the job via cluster mode, Airflow hands off the job to an available worker, therefore airflow has no knowledge of the spark job.

How can I address this issue?

apache-airflow airflow apache-spark
3 Answers

You can start leveraging the LivyOperator which will take care of monitoring the job, LivyOperator will poll the status of your Spark job at interval you will configure to poll. Example:

kickoff_streamer_task = LivyOperator(
    task_id='kickoff_streamer_task',
    dag=dag,
    livy_conn_id='lokori',
    file='abfs://data@amethiaprime.dfs.core.windows.net/user/draxuser/drax_streamer.jar',
    **polling_interval=60**,  # used when you want to pull the status of submitted job
    queue='root.ids.draxuser',
    proxy_user='draxuser',
    args=['10', '3000'],
    num_executors=4,
    conf={
        'spark.shuffle.compress': 'false',
        'master': 'yarn',
        'deploy_mode': 'cluster',
        'spark.ui.view.acls': '*'
    },
    class_name='com.apple.core.drax.dpaas.batch.DraxMatrixProducer',
    on_success_callback=livy_callback,
    on_failure_callback=_failure_callback
)

In the above example, polling_interval set to 60 seconds, it will keep on polling the status of your job at the 60s, it will make sure to give you the correct status of your job.

user_134603500
Related