Prometheus - Monitoring command output in container

user_8648858 0
Viewed 361

I need to monitoring a lot of legacy containers in my eks cluster that having a nfs mountpath. To map nfs directory in container i using nfs-client helm chart.

I need to monitor when my mountpath for some reason is lost, and the only way that i find to do that is exec a command in container.

#!/bin/bash

df -h | grep ip_of_my_nfs_server | wc -l

if the output above returns 1 i know that my nfs mountpath is ok.

Anybody knows some whay that monitoring an output script exec in container with prometheus?

Thanks!

prometheus kubernetes
1 Answers

As Matt has pointed out in the comments: first order of business should be to see if you can simply facilitate your monitoring requirement from node_exporter.

Below is a more generic answer on collecting metrics from arbitrary shell commands.

Prometheus is a pull-based monitoring system. You configure it with "scrape targets": these are effectively just HTTP endpoints that expose metrics in a specific format. Some target needs to be alive for long enough to allow it to be scraped.

The two most obvious options you have are:

  • Wrap your logic in a long-running process that exposes this metric on an HTTP endpoint, and configure it as a scrape target
  • Spin up an instance of pushgateway, and configure it as a scrape target , and have your command push its metrics there

Based on the little information you provided, the latter option seems like the most sane one. Important and relevant note from the README:

The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus. Since these kinds of jobs may not exist long enough to be scraped, they can instead push their metrics to a Pushgateway. The Pushgateway then exposes these metrics to Prometheus.

Your command would look something like:

#!/bin/bash

printf "mount_path_up %d" $(df -h | grep ip_of_my_nfs_server | wc -l) | curl --data-binary @- http://pushgateway.example.org:9091/metrics/job/some_job_name
user_35212430
Related