I don't understand how Datadog is aggregating my metrics

user_17800641 0
Viewed 914

I am trying to create a widget on my Datadog dashboard but having a hard time understanding what is going on. I have a metric that is a counter for successful queries and I want to get a time series that graphs the total count within a 1min period over time in order to visualize any significant drops in the count that would signify a problem with a server handling those queries. The query that I've tried using is something like this:

sum:count_metric{host}.as_count().rollup(sum, 60)

The sum at the start of the query, I believe, should be irrelevant because these metrics are only pulled from a single source. What I don't understand is as I change the display time (not sure how to word this better, but it's where I change time so the graph shows the metric over different periods of time e.g. the past 15min or past 4 hours etc. not the rollup period) the count that gets graphed changes too. The shorter the display time the lower the count. I am under the assumption that rollup() would aggregate the count by 1 minute so there shouldn't be any changes over different display times, but clearly that's not what's happening because when I display the past 15min the count is around 30, but if I display the past 1 day the count is around 600. What I want to see is the count hovering around 30 regardless of the display time. Another point of confusion is that when I display this timeseries using lines, the count is lower than if I were to use bars.

What am I misunderstanding about how Datadog calculates it's metrics (or just metrics in general) and how would I get the desired graph?

datadog
2 Answers

Take a look at the enforced vs custom rollup intervals. This is the key part here:

A custom .rollup() function can be used to enforce the type of time aggregation applied (avg, min, max, count, or sum) and the time interval to rollup. However, if a custom .rollup() function is applied and uses a smaller time interval than the Datadog limit, the Datadog limit is used instead while still using the specified rollup method.

So you won't be able to get the granularity you are after on longer timeframes.

For your use case, take a look at anomaly monitors and see if they will work.

user_4010960

If you want the values to be the same regardless of time window, then you need to normalize it as a per-second rate (instead of a count). You can then scale as needed if you still want to see average per-minute rates.

In my opinion, the 'as_count' feature is more trouble than it's worth if you are ever going to be changing the time window length. It just confuses folks.

user_34561370
Related