Apache Flink showing custom metrics in UI, but prometheus metrics reporter not scraping them

user_16610003 0
Viewed 368

I am working on sending custom app metrics to prometheus via the Prometheus Flink Metrics Reporter. The metrics are correctly created since I am able to accurately see them in the flink dashboard. I configured the prometheus metrics reporter similar to found here. When I curl to the prometheus endpoint (curl http://localhost:9090/api/v1/metrics), I am only able to see the cluster metrics and not the custom metrics I am creating. I suspect this issue has to do with how I configured the Prometheus Flink Metrics Reporter since when I try to visit http://localhost:9090, there is no UI and just a list of the cluster metrics mentioned above.

flink job code to create metrics(visible in Flink UI):

this.anomalyCounter = getRuntimeContext.getMetricGroup.addGroup("metric1").counter("counter")

flink-conf.yaml:

metrics.reporters: prom
metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter
metrics.reporter.prom.port: 9090

promethus.yml:

scrape_configs:
  - job_name: 'flink'
    static_configs:
      - targets: ['localhost:9090']

Is there anything I am missing in the configuration? Why are my cluster metrics reaching prometheus and not my custom ones?

prometheus apache-flink metrics
1 Answers

Hi @sarvad123 probably (depending on your Flink version) you should add flink-metrics-prometheus-{version}.jar in the /lib folder.

user_100601340
Related