Divide different node_exporter memory types by the total memory using group_left

user_3409092 0
Viewed 209

I would like to create a graph of the proportion of total memory consumed by each of the following for a given instance:

  • node_memory_MemFree_bytes{instance="$instance"}
  • node_memory_MemAvailable_bytes{instance="$instance"}
  • node_memory_Buffers_bytes{instance="$instance"}
  • node_memory_Cached_bytes{instance="$instance"}

I know I can create a single graph from 4 separate queries where each metric is divided by node_memory_MemTotal_bytes{instance="$instance"}:

  • Query A: node_memory_MemFree_bytes{instance="$instance"} / node_memory_MemTotal_bytes{instance="$instance"}
  • Query B: node_memory_MemAvailable_bytes{instance="$instance"} / node_memory_MemTotal_bytes{instance="$instance"}
  • Query C: node_memory_Buffers_bytes{instance="$instance"} / node_memory_MemTotal_bytes{instance="$instance"}
  • Query D: node_memory_Cached_bytes{instance="$instance"} / node_memory_MemTotal_bytes{instance="$instance"}

I feel like there should be a terser way of doing this with group_left since the right hand side is the same for each query and the left hand side can be reduced to a single query:

  • Left: {__name__=~"node_memory_(MemFree|MemAvailable|Buffers|Cached)_bytes",instance="$instance"}
  • Right: node_memory_MemTotal_bytes{instance="$instance"}

If I understand the "Operators" docs and this blog post correctly, I should be able to use group_left to do this in a single query. However, I've been unable to do so.

I reason that since the left and right side of the equation have exactly the same labels excluding __name__, I shouldn't need to use ignoring or on for the group_left operator:

{__name__=~"node_memory_(MemFree|MemAvailable|Buffers|Cached)_bytes",instance="$instance"}
/
on() group_left node_memory_MemTotal_bytes{instance="$instance"}

This query gives me an error:

"multiple matches for labels: grouping labels must ensure unique matches"

The same is true if I use ignoring() or any other combination of on(<labels>) and ignoring(<labels>). I can't wrap my head around what I'm missing here, is it a problem that the left and right don't have the same __name__? Am I approaching this problem incorrectly?

This operation is incredibly simple in Graphite using the divideSeries(dividendSeriesList, divisorSeries) function, surely I can do the same with PromQL.

Any help would be appreciated, thank you.

prometheus-node-exporter prometheus promql
1 Answers

PromQL strips metric names before matching time series on the left and right side of binary operator such as / - see these docs. In your case time series on the left side differ only by metric names, so PromQL sees series with identical sets of labels on the left side. The solution is to copy metric names for left-hand side time series into another label before applying the / operator and then ignoring this label during the / operation. This can be done with label_join or label_replace functions:

label_join({__name__=~"node_memory_(MemFree|MemAvailable|Buffers|Cached)_bytes",instance="$instance"}, "metric_name", "", "__name__")
/
ignoring(metric_name) group_left() node_memory_MemTotal_bytes{instance="$instance"}
user_2749370
Related