How to use prometheus operator service monitor to monitor the external service that is not running in the kubernetes pod

user_15450599 0
Viewed 2854

I am successfully configure my service monitor to monitor the API that provide metrics that is running in the kubernetes pod. However, I also would like to add external service to my service monitor target. This external service is arangoDB oasis exporter metrics (https://www.youtube.com/watch?v=c8i7K4HUPF4&t=554s). And this service is not running in kubernetes container. Here are my configuration files concerned:

  1. /helm/charts/prometheus-xxx/templates/service_monitor.tpl
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: {{ template "jobs-manager-servicemonitor.fullname" . }}
  # Change this to the namespace the jobs-manager-servicemonitor instance is running in
  namespace: {{ .Values.serviceMonitor.namespace }}
  labels:
    serviceapp: {{ template "jobs-manager-servicemonitor.name" . }}
    release: "{{ .Release.Name }}"
spec:
  selector:
    matchLabels:
      # Targets jobs-manager service
      app.kubernetes.io/instance: {{ .Values.instance.name }}
  endpoints:
    - port: {{ .Values.service.metricsPort.name }}
      interval: {{ .Values.serviceMonitor.interval }}
      {{- if .Values.serviceMonitor.scrapeTimeout }}
      scrapeTimeout: {{ .Values.serviceMonitor.scrapeTimeout }}
      {{- end }}
  namespaceSelector:
    matchNames:
    - {{ .Values.Namespace }}
  1. /helm/charts/prometheus-xxx/Chart.yaml
apiVersion: v1
appVersion: "1.0.0"
description: Prometheus Service monitor, customized
name: jobs-manager-servicemonitor
version: 1.0.1
  1. /helm/charts/prometheus-xxx/templates/_helpers.tpl
{{/*
Expand the name of the chart.
*/}}
{{- define "jobs-manager-servicemonitor.name" -}}
{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" }}
{{- end }}

{{/*
Create a default fully qualified app name.
We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
If release name contains chart name it will be used as a full name.
*/}}
{{- define "jobs-manager-servicemonitor.fullname" -}}
{{- if .Values.fullnameOverride -}}
{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- $name := default .Chart.Name .Values.nameOverride -}}
{{- if contains $name .Release.Name -}}
{{- .Release.Name | trunc 63 | trimSuffix "-" -}}
{{- else -}}
{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
{{- end -}}
{{- end -}}
{{- end -}}
  1. /helm/charts/prometheus-xxx/values.yaml
serviceMonitor:
  enabled: false
  namespace: prometheus
  interval: 10s
  scrapeTimeout: 10s

service:
  metricsPort:
    name: http

instance:
  name: jobs-manager
  
Namespace: test1

Do you have any suggestion on how to add the external service that is not running in kubernetes pod into the target of service monitor? Thank you very much in advance.

----------UPDATE POST----------

Here are my new config files in the charts template arangodb-servicemonitor:

  1. /helm/charts/arangodb-servicemonitor/templates/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: arangodb
  namespace: prometheus
  labels:
    app: arangodb
    release: prometheus
spec:
  type: ClusterIP
  externalName: xxxxx.arangodb.cloud:xxxx
  ports:
    - name: metrics
      port: 9000
      targetPort: 9000
      protocol: TCP
    - bearer_token: [ARANGODB_TOKEN]
    type: ExternalName
  1. /helm/charts/arangodb-servicemonitor/templates/endpoints.yaml
kind: Endpoints
apiVersion: v1
metadata:
  name: arangodb
  labels:
    app: arangodb
subsets:
- addresses:
  - ip: xxxxx.arangodb.cloud:xxxx
  ports: 
  - name: metrics
    port: 9000
    protocol: TCP
  1. /helm/charts/arangodb-servicemonitor/templates/service_monitor.tpl
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: arangodb-servicemonitor
  # Change this to the namespace the arangodb-servicemonitor instance is running in
  namespace: prometheus
  labels:
    serviceapp: arangodb-servicemonitor
    release: prometheus
spec:
  selector:
    # Targets arangodb service
    app: arangodb
  endpoints:
    # TO DO: use an array (List) of endpoints to monitor many endpoints
    - port: metrics
      interval: 30s
  namespaceSelector:
    matchNames:
    # TO DO: use an array (List) of endpoints to monitor many endpoints
    - default

But when I do terraform apply, I get this error message regarding the bearer_token that I added:

makeapplyERR

For info, I need to add this token to connect to ArangoDB external service:

connectArangoDB

prometheus-operator kubernetes-helm monitor service
1 Answers

Create the Kubernetes service and use the Prometheus ServiceMonitor same way.

create the K8s service

kind: Service
apiVersion: v1
metadata:
 name: arangoDB
spec:
 type: ClusterIP
 ports:
 - name: metrics
   port: 9000
   targetPort: 9000

create the Endpoint config of that Db in this prometheus should be able to access the IP of your Database

kind: Endpoints
apiVersion: v1
metadata:
 name: arangoDB
subsets:
 - addresses:
     - ip: IP of Database
   ports: 
     - name: metrics
       port: 9000

so nice service monitor will check the Kubernetes service and Kubernetes service will be pointing to the Endpoint and database and getting metrics from external service.

check endpoint service creation details at. : https://kubernetes.io/docs/concepts/services-networking/service/#services-without-selectors

user_55258240
Related