Issue getting Prometheus to monitor NGINX

Viewed 21

I've been trying to get Prometheus to monitor NGINX, the frustrating thing is it was working a few days ago and after changing the ServiceMonitor name to something more meaningful it hasn't worked since (I did change the name back and also restored backup files I made); obviously I've missed something important in how the objects fit together.

I have successfully configured nginx to export the metrics and can curl the NGINX endpoint using the k8s service and see the metrics (below).

NGINX is in the 'nginx-ingress' namespace and Prometheus is in the default namespace.

I have created a ServiceMonitor object that links to the service and a Prometheus object that links to the ServiceMonitor. Also I have created a service account with the correct Clusterrole and Clusterrolebinding for the Prometheus object.

Any help / advice appreciated in fixing this, please see my yaml files below:

k exec curl -n nginx-ingress -- curl nginx-prometheus-svc:9113/metrics
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  6764    0  6764    0     0  2711k      0 --:--:-- --:--:-- --:--:-- 3302k
# HELP nginx_ingress_controller_ingress_resources_total Number of handled ingress resources
# TYPE nginx_ingress_controller_ingress_resources_total gauge
nginx_ingress_controller_ingress_resources_total{class="nginx",type="master"} 0
nginx_ingress_controller_ingress_resources_total{class="nginx",type="minion"} 0
nginx_ingress_controller_ingress_resources_total{class="nginx",type="regular"} 0
# HELP nginx_ingress_controller_nginx_last_reload_milliseconds Duration in milliseconds of the last NGINX reload
# TYPE nginx_ingress_controller_nginx_last_reload_milliseconds gauge
nginx_ingress_controller_nginx_last_reload_milliseconds{class="nginx"} 139

Service:

apiVersion: v1
kind: Service
metadata:
  name: nginx-prometheus-svc
  labels:
    name: nginx-prometheus-svc # used by ServiceMonitor
  namespace: nginx-ingress # namespace where the pods / svc are located
spec:
  type: ClusterIP
  ports:
    - name: prometheus # IMPORTANT: the port must have a name for use in the ServiceMonitor
      port: 9113 # port used by Prometheus for scraping
      targetPort: 9113
  selector:
    app: nginx-ingress # label of the NGINX pods to be monitored

ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    name: heartbeat-kube-prometheus-nginx # Used by the Prometheus object
  namespace: default # Namespace of the Prometheus operator
  name: heartbeat-kube-prometheus-nginx
spec:
  endpoints: # endpoints to connect to for scraping
  - path: /metrics
    port: prometheus
  namespaceSelector:
    matchNames:
    - nginx-ingress # Namespace of the svc
  selector:
    matchLabels:
      name: nginx-prometheus-svc # Label of the svc

Prometheus object:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus-scrape
  namespace: default
spec:
  serviceAccountName: prometheus # sa with correct rbac to access the resources to be monitored
  serviceMonitorSelector:
    matchLabels:
      name: heartbeat-kube-prometheus-nginx # Label of the service monitor

Service account + RBAC:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  ----
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
  ---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default

Excerpt from the NGINX pod:

  annotations:
    prometheus.io/port: "9113"
    prometheus.io/scheme: http
    prometheus.io/scrape: "true"
  creationTimestamp: "2022-09-14T11:56:18Z"
  generateName: nginx-ingress-65c87cbfbc-
  labels:
    app: nginx-ingress
    pod-template-hash: 65c87cbfbc
  name: nginx-ingress-65c87cbfbc-5lbcv
  namespace: nginx-ingress
0 Answers
Related