Monitor application metric to grafana with grafana agent / prometheus

user_4034179 0
Viewed 1605

Description

Being new to the grafana and prometheus world, I am struggeling to add custom metrics from my laravel php cli application to grafana cloud - preferably via grafana agent.

Situation

I am using grafana cloud with their grafana agent on a linux server that is running a laravel php worker without a web server. The grafana agent is running with node_exporter integration. I have already tried to find some documentation on how to add a custom exporter or adding a scraper to gather information. What I have currently found is that the agent will be somehow (?) querying an HTTP endpoint and parse the response (which format?) to post it to the grafa cloud endpoint (prometheus push gateway as far as I understood).

I did not find a documentation on how to write a custom exporter for the grafana agent, since I am running a php worker thread without an http endpoint on that server. Exposing those information on an endpoint is doable but feels wrong, isn't it? Basically I'd like to do a 'php artisan mypackage:metrics' and let that call generate the correct output which then is used by the agent to post to grafana.

Questions

  • How can I write my custom exporter that is queried by the grafana agent?
  • What is the correct data format?
  • If grafana agent exporter is not the right direction, how can scraping work?

What I've tried

  • regarding the data structure / format

According to [1], I tried to create my custom metrics like the following - correct?

# TYPE mynamespace_some_metric counter
mynamespace_some_metric 42
  • regaring posting to grafana cloud

Grafana cloud offers a remote write endpoint with a URL like https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push with bearer authentication token. I tried to send a POST request to that endpoint with the data above but only receive a 400 bad request respone saying snappy: corrupt input.

So I think there are some basic understandings missing and hope to get a kick in the right direction...

[1] - https://www.metricfire.com/blog/prometheus-pushgateways-everything-you-need-to-know/#strongSending-Metricsstrong

Solution

Thanks to @anemyte i came up with this solution, that is working great now:

  1. Create a Controller, that renders the desired output with the promphp/prometheus_client_php package like so:

     $registry = new CollectorRegistry(new InMemory());

 $counter = $registry->getOrRegisterGauge('test', 'some_counter', 'it sets', ['type']);
 $counter->set(rand(1, 99), ['blue']);

 $renderer = new RenderTextFormat();
 $result = $renderer->render($registry->getMetricFamilySamples());

 header('Content-type: ' . RenderTextFormat::MIME_TYPE);
 echo $result;

  2. Register a route /metrics that uses the Controller above.

  3. Configure the /etc/grafana-agent.yaml and add a config in the prometheus yaml section:

  prometheus:   
    configs:
      - name: mycustomwebsite
        scrape_configs:
          - job_name: default
            static_configs:
                - targets: ['www.mywebsite.de:80']
        remote_write:
        - basic_auth:
            password: YOURPASSWORD
            username: YOURUSERNAME
          url: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push
      - name: integrations
        remote_write:
        - basic_auth:
            password: YOURPASSWORD
            username: YOURUSERNAME
          url: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push
  1. Restart the agent and your metrics are ready to use.
grafana prometheus
2 Answers

Grafana Agent works with the same metric format Prometheus does. It is focused on scraping metrics instead of Prometheus and pushing them (remote_write) to the Prometheus instance that Grafana Cloud hosts for you. This is mentioned in the list of product features.

You can use Prometheus PHP library to create metrics and avoid troubles with the raw format. The best practices are also applicable.

Once you are done with creating metrics, you need to instruct the agent to scrape them from your server. Use these docs (one, two) for reference.

user_113445020

Solution

Thanks to @anemyte i came up with this solution, that is working great now.

  1. Create a Controller, that renders the desired output with the promphp/prometheus_client_php package like so:

     $registry = new CollectorRegistry(new InMemory());

 $counter = $registry->getOrRegisterGauge('test', 'some_counter', 'it sets', ['type']);
 $counter->set(rand(1, 99), ['blue']);

 $renderer = new RenderTextFormat();
 $result = $renderer->render($registry->getMetricFamilySamples());

 header('Content-type: ' . RenderTextFormat::MIME_TYPE);
 echo $result;

  2. Register a route /metrics that uses the Controller above.

  3. Configure the /etc/grafana-agent.yaml and add a config in the prometheus yaml section:

  prometheus:   
    configs:
      - name: mycustomwebsite
        scrape_configs:
          - job_name: default
            static_configs:
                - targets: ['www.mywebsite.de:80']
        remote_write:
        - basic_auth:
            password: YOURPASSWORD
            username: YOURUSERNAME
          url: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push
      - name: integrations
        remote_write:
        - basic_auth:
            password: YOURPASSWORD
            username: YOURUSERNAME
          url: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push
  1. Restart the agent and your metrics are ready to use.
user_40341790
Related