Prometheus Monitoring for Remote network

user_17497183 0
Viewed 1189

I am very new at Prometheus. I am currently a working student and my task is to build a monitoring system for servers running on client side (not accessible by us). I have installed node exporter and slave Prometheus serves at the client side (Ubuntu VM, Kubernetes nodes). We have a centralised master Prometheus server and Grafana hosted on AWS, but it cannot reach the client Prometheus server.

Is there any way the slave Prometheus can push the metrics to the master Prometheus server without giving them access to each other? I tried other ways like installing Graphite remote adapter and configuring the slave Prometheus remote write, but there was some issue with the adapter. Will Prometheus Telegraf and influxdb work in this case?

I am sorry if I framed my question wrong.

UPDATE:

This is my MASTER Prometheus server's .yml file: (this one has remote write enabled). This server is the central one, so it is supposed to get all metrics from all remote client Prometheus servers.

global:
  scrape_interval:     15s 
  evaluation_interval: 15s 
  
scrape_configs:
  - job_name: 'Scoutastic-prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['10.0.4.175:9090']

This is a client Prometheus server's (unreachable) .yml file. This has remote write url of master Prometheus.

gloabal:
  scrape_interval:     5m
  evaluation_interval: 5m

remote_write:
 - url: http://10.0.4.175:9090/api/v2/write
   write_relabel_configs:
   - source_labels: [_node_]
     action: keep

   queue_config:
     capacity: 10000
     max_shards: 10
     min_shards: 2
     max_samples_per_send: 5000
     batch_send_deadline: 30m
     min_backoff: 50ms
     max_backoff: 1s

scrape_configs:
   - job_name: 'node'
     scrape_interval: 5s
     static_configs:
      - targets: ['localhost:9100']

I don’t have any configuration other than this. Thank you in advance.

prometheus
1 Answers

Prometheus can push (remote_write) and pull (remote_read) metrics from other services and Prometheus instances. This is primarily intended for a long term storage; if you want to alert using that data, it may require tuning as the data comes with a delay.

To make Prometheus push metrics to another Prometheus you need to start the receiving one with this flag:

--enable-feature=remote-write-receiver

Then configure remote_write on your sender instance:

remote_write:
- url: http://your-main-prometheus.com/api/v1/write
user_113445020
Related