Custom gauge for prometheus Go SDK

Viewed 29

I have a APP that monitors some external jobs (among other things). This does this monitoring every 5 Mins

I'm trying to create a prometheus gauge to get the count of currently running jobs.

Here is how I declared my gauge

JobStats= promauto.NewGaugeVec(
        prometheus.GaugeOpts{
            Namespace:   "myapi",
            Subsystem:   "app",
            Name:        "job_count",
            Help:        "Current running jobs in the system",
            ConstLabels: nil,
        },
        []string{"l1", "l2", "l3"},
    )

in the code that actually counts the jobs I do

metrics.JobStats.WithLabelValues(l1,l2,l3).add(float64(jobs_cnt))

when I query the /metrics endpoint I get the number

The thing is, this metrics only keeps increasing. If I restart the app this get resets to zer & again keeps increasing

I'm using grafana to graph this in a dashboard.

My question is

  1. Get the graph to show the actual number of jobs (instead of ever increasing line)?
  2. Should this be handled in code (like setting this to zero before every collection?) or in grafana?
0 Answers
Related