What is the correct prometheus URL to be used by prometheus-adapter

user_10748166 0
Viewed 3469

I have successfully deployed

using default configuration with slight customization.

I can access prometheus, grafana and alertmanager, query metrics and see fancy charts.

But prometheus-adapter keeps complaining on startup that it can't access/discover metrics:

I0326 08:16:52.266095       1 adapter.go:98] successfully using in-cluster auth
I0326 08:16:52.330094       1 dynamic_serving_content.go:111] Loaded a new cert/key pair for "serving-cert::/var/run/serving-cert/tls.crt::/var/run/serving-cert/tls.key"
E0326 08:16:52.334710       1 provider.go:227] unable to update list of all metrics: unable to fetch metrics for query "{namespace!=\"\",__name__!~\"^container_.*\"}": bad_response: unknown response code 404

I've tried various prometheus URLs in the prometheus-adapter Deployment command line argument but the problem is more or less the same.

E.g. some of the URLs I've tried are

--prometheus-url=http://prometheus-operated.prom.svc:9090

--prometheus-url=http://prometheus-kube-prometheus-prometheus.prom.svc.cluster.local:9090

There are the following services / pods running:

$ kubectl -n prom get pods
NAME                                                     READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-kube-prometheus-alertmanager-0   2/2     Running   0          16h
prometheus-adapter-76fcc79b7b-7xvrm                      1/1     Running   0          10m
prometheus-grafana-559b79b564-bh85n                      2/2     Running   0          16h
prometheus-kube-prometheus-operator-8556f58759-kl84l     1/1     Running   0          16h
prometheus-kube-state-metrics-6bfcd6f648-ms459           1/1     Running   0          16h
prometheus-prometheus-kube-prometheus-prometheus-0       2/2     Running   1          16h
prometheus-prometheus-node-exporter-2x6mt                1/1     Running   0          16h
prometheus-prometheus-node-exporter-bns9n                1/1     Running   0          16h
prometheus-prometheus-node-exporter-sbcjb                1/1     Running   0          16h

$ kubectl -n prom get services
NAME                                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                     ClusterIP   None           <none>        9093/TCP,9094/TCP,9094/UDP   16h
prometheus-adapter                        ClusterIP   10.0.144.45    <none>        443/TCP                      16h
prometheus-grafana                        ClusterIP   10.0.94.160    <none>        80/TCP                       16h
prometheus-kube-prometheus-alertmanager   ClusterIP   10.0.0.135     <none>        9093/TCP                     16h
prometheus-kube-prometheus-operator       ClusterIP   10.0.170.205   <none>        443/TCP                      16h
prometheus-kube-prometheus-prometheus     ClusterIP   10.0.250.223   <none>        9090/TCP                     16h
prometheus-kube-state-metrics             ClusterIP   10.0.135.215   <none>        8080/TCP                     16h
prometheus-operated                       ClusterIP   None           <none>        9090/TCP                     16h
prometheus-prometheus-node-exporter       ClusterIP   10.0.70.247    <none>        9100/TCP                     16h

kubectl -n kube-system get deployment/metrics-server
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
metrics-server   1/1     1            1           15d

Prometheus-adapter helm chart gets deployed using the following values:

prometheus:
  url: http://prometheus-kube-prometheus-prometheus.prom.svc.cluster.local
certManager:
  enabled: true

What is the correct value for --prometheus-url for prometheus-adapter in my setup ?

prometheus kubernetes
3 Answers

The problem is related to the additional path used to expose prometheus via Ingress.

I was using additional path prefix /monitoring/prometheus/ for my ingress configuration.

The solution is to tell prometheus-adapter too, that prometheus is accessible incl. this path prefix. Thus the following makes prometheus-adapter happy :

--prometheus-url=http://prometheus-kube-prometheus-prometheus.prom.svc.cluster.local:9090/monitoring/prometheus/

And now I can see custom metrics when executing

kubectl get --raw /apis/custom.metrics.k8s.io/v1beta1

Thank you "rock'n rolla" for giving some hints !

user_107481660

I'm using both helm charts (kube-prometheus-stack and prometheus-adapter).

additional path prefix that works for me is "/", but, prometheus url must be with the name of your helm-install parameter for stack ("helm install "). I'm using "prostack" as stack name. So finally, it works for me:

helm install <adapter-name> -n <namespace> --set prometheus.url=http://prostack-kube-prometheus-s-prometheus.monitoring.svc.cluster.local --set prometheus.port=9090 --set prometheus.path=/
user_155856040

I struggled with the same issue. When i installed my prometheus server with Helm via the community chart.

I got a message like this:

The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster: my-tag-prometheus-server.carbon.svc.cluster.local

Please note that is says the service is accessible on port 80 and not 9090 which i had not noticed.

So in my values.yaml file for the prometheus-adapter chart i specified port 80 and it worked (instead of the standard 9090)

 28 # Url to access prometheus
 29 prometheus:
 30   # Value is templated
 31   url: http://my-tag-prometheus-server.carbon.svc.cluster.local
 32   port: 80
 33   path: ""
user_93007420
Related