Ingress routing rules to access prometheus server

user_627855 0
Viewed 2483

I have deployed prometheus server(2.13.1) on kubernetes(1.17.3), I am able to access it on http://my.prom.com:9090

But i want to access it on http://my.prom.com:9090/prometheus so i added following ingress rules but its not working

First Try:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/app-root: /prometheus
  name: approot
  namespace: default
spec:
  rules:
  - host: my.prom.com
    http:
      paths:
      - backend:
          serviceName: prometheus-svc
          servicePort: 9090
        path: /

This results in 404 error

Second Try:

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  name: rewrite
  namespace: default
spec:
  rules:
  - host: my.prom.com
    http:
      paths:
      - backend:
          serviceName: prometheus-svc
          servicePort: 9090
        path: /prometheus(/|$)(.*)

Now when i access URL http://my.prom.com:9090/prometheus in browser it get changed to http://my.prom.com:9090/graph and show 404 error

prometheus kubernetes-ingress kubernetes
3 Answers

Prometheus is not aware of what you are trying to achieve and that's why it's redirecting to unknown destination.

You have to tell prometheus to accept traffic on the new path as can be seen here and here.

Highlight to the second link, you have to include - "--web.route-prefix=/" and - "--web.external-url=http://my.prom.com:9090/prometheus" in your prometheus deployment.

Then I had to modify the prometheus deployment to accept traffic on the new path (/prom). This was covered in the Securing Prometheus API and UI Endpoints Using Basic Auth documentation:

In your env it should look like this:

> grep web deploy.yaml 
            - "--web.enable-lifecycle"
            - "--web.route-prefix=/"
            - "--web.external-url=http://my.prom.com:9090/prometheus"
user_121535760

I was experiencing this issue while deploying Prometheus via the community Helm chart and figured it may be helpful to share my findings with others here: https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus

My override values.yaml looks like this:

  server:
    prefixURL: /
    baseURL: https://my-cluster-external-hostname/prometheus
    ingress:
      enabled: true
      ingressClassName: nginx
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/$2"
      path: "/prometheus(/|$)(.*)"
      hosts:
        - my-cluster-external-hostname
      tls:
        - secretName: cluster-tls-secret
          hosts:
            - my-cluster-external-hostname
    service:
      servicePort: 9090

Now the resulting deployment spec for prometheus-server ends up with these args (note the order and specifically that --web.route-prefix is at the top):

      --web.route-prefix=/
      --storage.tsdb.retention.time=15d
      --config.file=/etc/config/prometheus.yml
      --storage.tsdb.path=/data
      --web.console.libraries=/etc/prometheus/console_libraries
      --web.console.templates=/etc/prometheus/consoles
      --web.enable-lifecycle
      --web.external-url=https://my-cluster-external-hostname/prometheus

This does NOT work as the /-/healthy endpoint results in a 404 (from kubectl describe pod prometheus-server):

  Warning  Unhealthy  9s (x9 over 49s)  kubelet            Readiness probe failed: HTTP probe failed with statuscode: 404
  Warning  Unhealthy  9s (x3 over 39s)  kubelet            Liveness probe failed: HTTP probe failed with statuscode: 404

After much trial and error, I realized the order of those arguments seemed to matter so I changed my Helm chart values.yaml as follows:

  server:
    # prefixURL: /   # <-- commented out
    # baseURL: https://my-cluster-external-hostname/prometheus  # <-- commented out
    extraFlags:  # <-- added this section to specify my args manually
      - web.enable-lifecycle
      - web.route-prefix=/
      - web.external-url=https://my-cluster-external-hostname/prometheus
    ingress:
      enabled: true
      ingressClassName: nginx
      annotations:
        nginx.ingress.kubernetes.io/rewrite-target: "/$2"
      path: "/prometheus(/|$)(.*)"
      hosts:
        - my-cluster-external-hostname
      tls:
        - secretName: cluster-tls-secret
          hosts:
            - my-cluster-external-hostname
    service:
      servicePort: 9090

The resulting deployment from this values.yaml puts the arguments in the apparently proper order which makes the health check endpoint available (internally) and allows accessing Prometheus from outside the cluster. Notice where --web.route-prefix is located now.

      --storage.tsdb.retention.time=15d
      --config.file=/etc/config/prometheus.yml
      --storage.tsdb.path=/data
      --web.console.libraries=/etc/prometheus/console_libraries
      --web.console.templates=/etc/prometheus/consoles
      --web.enable-lifecycle
      --web.route-prefix=/
      --web.external-url=https://my-cluster-external-hostname/prometheus

I also submitted a bug to the community Prometheus chart: https://github.com/prometheus-community/helm-charts/issues/1594

user_12217180

add in deploy-prometheus.yml

args:
    - --web.enable-lifecycle
    - --web.route-prefix=/
    - --web.external-url=https://localhost:9090/prometheus/

In VirtualService the Prometheus

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: prometheus-vs
  namespace: istio-system
spec:
  hosts:
  - "*"
  gateways:
  - prometheus-gateway
  http:
    - match:
      - uri:
          prefix: /prometheus/
      rewrite:
        uri: /
      route:
      - destination:
          host: prometheus
          port:
            number: 9090
user_37402440
Related