I'm having trouble setting up monitor that will alert me when an event hasn't happened since some period of time following another event. Basically, for a given task in my application, I have a log that indicates a state of "running" and another log that indicates a state of "finished". From these logs, I've defined two custom metrics in datadog. I'm trying to set up a monitor that will alert me when a task has not finished within 2 hours of when it started running. So for example, if the running metric is observed at 2:00, the monitor shouldn't alert for the absence of the finished until 4:00. If the finished metric is observed before 4:00, the monitor will not alert for this task.
The way that I've tried implementing this is by using a threshold monitor, and subtracting the count of my running metric from the count of my finished metric. However, what's challenging here is the time-delta piece.
I've tried using the delay evaluation (delaying by 2 hrs), however, at the time when it starts evaluating, it will only take into account the first metric. It basically, just slides the window back.