Every Prometheus job as a data source in Grafana

Question

We have a Prometheus, a Grafana service and multiple exporters. This is prometheus.yaml config:

scrape_configs: 
  - job_name: app1
    scrape_interval: 30s 
    scheme: http
    static_configs: 
      - targets: ['x.y.z.10:7380'] 

  - job_name: app2
    scrape_interval: 30s 
    scheme: http
    static_configs: 
      - targets: ['x.y.z.11:7380'] 

  - job_name: app3
    scrape_interval: 30s 
    scheme: http
    static_configs: 
      - targets: ['x.y.z.12:7380']

from Grafana's point of view, those scrapes will be accessed as a single data source. However, to make different dashboards for every application (app1, app2 and app3) we separated them by appending job filter {job="appX"} to queries/metrics.

This makes two problems, the first one is, we have to add {job="appX"} after every metric. The second one is, the list of metrics are populated of unrelated items from another jobs.

Is there any method in Prometheus or Grafana to make each job as separated as a data source? In which we don't have to add {job="appX"} to metrics. Let's say we don't have any access to another job's metrics in a specific dashboard.

user_2761985 · Answer

A good practice is to prefix your metric names with your application name as 'namespace'. e.g.:

appx_mycounter
appy_mycounter
appz_mycounter

This would enable you to ignore the job label completely and the typeahead in Grafana would work much better.

user_10681033 · Answer

No, there is no easy way to separate jobs into different data sources. The proper way imo you can handle it is to create a dashboard variable for the job_name which can be dynamically scraped, and have 1 dashboard which you can chose to show either job's metrics by adding {job="${variable}"} filter. If you don't want to add label filters after all the metrics you can use to add a pre/sufix to all the metrics to make them for ex. app1_uptime https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config

user_274937 · Answer

I don't know a straightforward way to create per-job datasource in Grafana for metrics stored in Prometheus. But this can be easily done with VictoriaMetrics - just add extra_filters[]={job="appN"} query arg at custom query parameters input field per each datasource in Grafana. See the following additional information:

Every Prometheus job as a data source in Grafana

3 Answers