Difference between two metrics with time offset in PromQL

user_569916 0
Viewed 2605

I have a metric (gauge) updated, which has module label that's set to a name of a module. Modules set it to UTC now after they've performed a successful "update" (whatever it means). One of the modules (call it "main") must perform an update not later than, say, 10 minutes after any of other modules are updated.

Attempt #1 Extract max value of all other metrics from "main" metric:

(max(updated{module="main"}) - max(updated{module!~"main"}))

It partially works. It fails in those periods of time where an "other" metric has updated, but "main" has not yet:

enter image description here

Attempt #2

Select "other" metrics with 10 minutes offset, so we always compare "main" update time with the latest "other" metric update time taken at now - 10m moment, which gives "main" module 10 minutes to do its work:

(max(updated{module="main"}) - max(updated{module!~"main"} offset 10m))

The result is not good either:

enter image description here

Now we have a huge delay between updates (8K seconds in this case) because we subtract a too old "other" update time from the current "main" time.

I'm new to PromQL, so chances are high I'm missing some simple fundamental approach.

prometheus promql
1 Answers

Try the following PromQL query:

(max(updated{module="main"}) < max(updated{module!="main"}))
  and
(time() - max(updated{module!="main"}) > 10*60)

It works in the following way:

  1. The max(updated{module="main"}) < max(updated{module!="main"}) returns non-empty result when the main module isn't updated yet after the last update across non-main modules.
  2. The time() - max(updated{module!="main"}) > 10*60 returns non-empty result when the duration since the last module update exceeds 10 minutes.

When these queries are combined with and operator, then the resulting query returns non-empty result if the main module isn't updated yet after 10 minutes since the last update of non-main module. See docs for and operator and docs for time().

user_2749370
Related