Send an alert when a dag did not run google cloud

user_3106963 0
Viewed 373

I have a DAG in Airflow where the run is not scheduled, but triggered by an event. I would like to send an alert when the DAG did not run in the last 24 hours. My problem is I am not really sure which tool is the best for the task.

I tried to solve it with the Logs Explorer, I was able to write a quite good query filtering by the textPayload, but it seems that tool is designed to send the alert when a specific log is there, not when it is missing. (Maybe I missed something?)

I also checked Monitoring where I could set up an Alert when logs are missing, however in this case I was not able to write any query where I can filter logs by textPayload.

Thank you in advance if you can help me!

google-cloud-monitoring google-cloud-logging airflow
2 Answers

You could set up a separate alert DAG that notifies you if other DAGs haven't run in a specified amount of time? To get the last runtime of a DAG, use something like this:

from airflow.models import DagRun

dag_runs = DagRun.find(dag_id=dag_id)
dag_runs.sort(key=lambda x: x.execution_date, reverse=True)

Then you can use dag_runs[0] and compare with the current server time. If the date difference is greater than 24h, raise an alert.

user_182997500

I was able to do it in the monitoring. I did not need the filtering query which I used in the Logs Explorer. I needed to create an Alerting Policy, filtered by workflow_name, task_name and location. In the configure trigger section I was able to choose "Metric absence" with a 1 day absence time, so I resolved my old query with this.

Of course, it could be solved with setting up a new DAG, but setting up an Alerting Policy seems more easier.

user_31069630
Related