I've been trying to get Prometheus to monitor NGINX, the frustrating thing is it was working a few days ago and after changing the ServiceMonitor name to something more meaningful it hasn't worked since (I did change the name back and also restored backup files I made); obviously I've missed something important in how the objects fit together.
I have successfully configured nginx to export the metrics and can curl the NGINX endpoint using the k8s service and see the metrics (below).
NGINX is in the 'nginx-ingress' namespace and Prometheus is in the default namespace.
I have created a ServiceMonitor object that links to the service and a Prometheus object that links to the ServiceMonitor. Also I have created a service account with the correct Clusterrole and Clusterrolebinding for the Prometheus object.
Any help / advice appreciated in fixing this, please see my yaml files below:
k exec curl -n nginx-ingress -- curl nginx-prometheus-svc:9113/metrics
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 6764 0 6764 0 0 2711k 0 --:--:-- --:--:-- --:--:-- 3302k
# HELP nginx_ingress_controller_ingress_resources_total Number of handled ingress resources
# TYPE nginx_ingress_controller_ingress_resources_total gauge
nginx_ingress_controller_ingress_resources_total{class="nginx",type="master"} 0
nginx_ingress_controller_ingress_resources_total{class="nginx",type="minion"} 0
nginx_ingress_controller_ingress_resources_total{class="nginx",type="regular"} 0
# HELP nginx_ingress_controller_nginx_last_reload_milliseconds Duration in milliseconds of the last NGINX reload
# TYPE nginx_ingress_controller_nginx_last_reload_milliseconds gauge
nginx_ingress_controller_nginx_last_reload_milliseconds{class="nginx"} 139
Service:
apiVersion: v1
kind: Service
metadata:
name: nginx-prometheus-svc
labels:
name: nginx-prometheus-svc # used by ServiceMonitor
namespace: nginx-ingress # namespace where the pods / svc are located
spec:
type: ClusterIP
ports:
- name: prometheus # IMPORTANT: the port must have a name for use in the ServiceMonitor
port: 9113 # port used by Prometheus for scraping
targetPort: 9113
selector:
app: nginx-ingress # label of the NGINX pods to be monitored
ServiceMonitor:
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
labels:
name: heartbeat-kube-prometheus-nginx # Used by the Prometheus object
namespace: default # Namespace of the Prometheus operator
name: heartbeat-kube-prometheus-nginx
spec:
endpoints: # endpoints to connect to for scraping
- path: /metrics
port: prometheus
namespaceSelector:
matchNames:
- nginx-ingress # Namespace of the svc
selector:
matchLabels:
name: nginx-prometheus-svc # Label of the svc
Prometheus object:
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: prometheus-scrape
namespace: default
spec:
serviceAccountName: prometheus # sa with correct rbac to access the resources to be monitored
serviceMonitorSelector:
matchLabels:
name: heartbeat-kube-prometheus-nginx # Label of the service monitor
Service account + RBAC:
apiVersion: v1
kind: ServiceAccount
metadata:
name: prometheus
----
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: prometheus
rules:
- apiGroups: [""]
resources:
- nodes
- nodes/metrics
- services
- endpoints
- pods
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources:
- configmaps
verbs: ["get"]
- apiGroups:
- networking.k8s.io
resources:
- ingresses
verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: prometheus
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: prometheus
subjects:
- kind: ServiceAccount
name: prometheus
namespace: default
Excerpt from the NGINX pod:
annotations:
prometheus.io/port: "9113"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
creationTimestamp: "2022-09-14T11:56:18Z"
generateName: nginx-ingress-65c87cbfbc-
labels:
app: nginx-ingress
pod-template-hash: 65c87cbfbc
name: nginx-ingress-65c87cbfbc-5lbcv
namespace: nginx-ingress