How to monitor external service in prometheus-operator

user_1780879 0
Viewed 9084

I am trying to monitor external service (which is exporter of cassandra metrics) in prometheus-operator. I installed prometheus-operator using helm 2.11.0. I installed it using this yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: tiller
  namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system

and these commands on my kubernetes cluster:

kubectl create -f rbac-config.yml
helm init --service-account tiller --history-max 200
helm install stable/prometheus-operator --name prometheus-operator --namespace monitoring

Next, basing on article: how monitor to an external service

I tried to do steps described in it. As suggested I created Endpoints, Service and ServiceMonitor with label for existing Prometheus. Here are my yaml files:

apiVersion: v1
kind: Endpoints
metadata:
  name: cassandra-metrics80
  labels:
    app: cassandra-metrics80
subsets:
- addresses:
  - ip: 10.150.1.80
  ports:
  - name: web
    port: 7070
    protocol: TCP

apiVersion: v1
kind: Service
metadata:
  name: cassandra-metrics80
  namespace: monitoring
  labels:
    app: cassandra-metrics80
    release: prometheus-operator
spec:
  externalName: 10.150.1.80
  ports:
  - name: web
    port: 7070
    protocol: TCP
    targetPort: 7070
  type: ExternalName

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: cassandra-metrics80
  labels:
    app: cassandra-metrics80
    release: prometheus-operator
spec:
  selector:
    matchLabels:
      app: cassandra-metrics80
      release: prometheus-operator
    namespaceSelector:
      matchNames:
      - monitoring
  endpoints:
  - port: web
    interval: 10s
    honorLabels: true

And in prometheus service discovery page I can see: Service Discovery

That this service is not active and all labels are dropped. I did a numerous things trying to fix this, like setting targetLabels. Trying to relabel the once that are discovered, as described here: prometheus relabeling But unfortunately nothing works. What could be the issue or how can I investigate it better?

prometheus-operator prometheus kubernetes-helm kubernetes
3 Answers

Ok, I found out that service should be in the same namespace as service monitor and endpoint, after that prometheus started to see some metrics from cassandra.

user_17808790

When using the kube-prometheus-stack helm chart, it can be done as follows

prometheus:
  prometheusSpec:
    additionalScrapeConfigs:
    - job_name: external
      metrics_path: /metrics
      static_configs:
        - targets:
          - <IP>:<PORT>
user_41857850

To be strict only "Endpoints" and "Service" should be in the same namespace. Additionally "Endpoints" and "Service" should have the same name as Lucas mentioned it before.

ServiceMonitor can be placed anywhere, it finds and scrapes SVC/Endpoint inside defined namespaces (namespaceSelector->matchNames) and matching all labels (selector->matchLabels):

    spec:
  selector:
    matchLabels:
      app: cassandra-metrics80
      release: prometheus-operator
  namespaceSelector:
    matchNames:
    - my-namespace

Furthermore now there is much more easier method to define additional scraping: https://github.com/prometheus-operator/prometheus-operator/blob/master/Documentation/additional-scrape-config.md The only drawback for the second one is that it requires pod restart after the change. Configuration based on Endpoint/Service/ServiceMonitor seem to be discovered and applied automatically.

user_142477570
Related