I followed this THIS tutorial. I stuck on chapter "INSTALL PROMETHEUS ADAPTER ON KUBERNETES"
I before installed prometheus and after adapter with this tut.
Loogs from adapter.
I1201 18:29:51.064522 1 shared_informer.go:204] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I1201 18:29:51.064793 1 tlsconfig.go:157] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubernetes" [] issuer="<self>" (2020-11-26 13:25:15 +0000 UTC to 2030-11-24 13:25:15 +0000 UTC (now=2020-12-01 18:29:51.064776377 +0000 UTC))
I1201 18:29:51.064996 1 tlsconfig.go:179] loaded serving cert ["serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key"]: "localhost@1606847390" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer="localhost-ca@1606847390" (2020-12-01 17:29:50 +0000 UTC to 2021-12-01 17:29:50 +0000 UTC (now=2020-12-01 18:29:51.064987277 +0000 UTC))
I1201 18:29:51.065289 1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1606847390" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1606847390" (2020-12-01 17:29:50 +0000 UTC to 2021-12-01 17:29:50 +0000 UTC (now=2020-12-01 18:29:51.065278277 +0000 UTC))
I1201 18:29:51.065471 1 tlsconfig.go:157] loaded client CA [0/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubernetes" [] issuer="<self>" (2020-11-26 13:25:15 +0000 UTC to 2030-11-24 13:25:15 +0000 UTC (now=2020-12-01 18:29:51.065459677 +0000 UTC))
I1201 18:29:51.065502 1 tlsconfig.go:157] loaded client CA [1/"client-ca::kube-system::extension-apiserver-authentication::client-ca-file,client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"]: "kubernetes" [] issuer="<self>" (2020-11-26 13:25:16 +0000 UTC to 2030-11-24 13:25:16 +0000 UTC (now=2020-12-01 18:29:51.065493477 +0000 UTC))
I1201 18:29:51.065715 1 tlsconfig.go:179] loaded serving cert ["serving-cert::/tmp/cert/apiserver.crt::/tmp/cert/apiserver.key"]: "localhost@1606847390" [serving] validServingFor=[127.0.0.1,localhost,localhost] issuer="localhost-ca@1606847390" (2020-12-01 17:29:50 +0000 UTC to 2021-12-01 17:29:50 +0000 UTC (now=2020-12-01 18:29:51.065704777 +0000 UTC))
I1201 18:29:51.065910 1 named_certificates.go:52] loaded SNI cert [0/"self-signed loopback"]: "apiserver-loopback-client@1606847390" [serving] validServingFor=[apiserver-loopback-client] issuer="apiserver-loopback-client-ca@1606847390" (2020-12-01 17:29:50 +0000 UTC to 2021-12-01 17:29:50 +0000 UTC (now=2020-12-01 18:29:51.065901277 +0000 UTC))
I1201 18:30:24.362727 1 httplog.go:90] GET /healthz: (2.2796ms) 200 [kube-probe/1.16+ 10.1.0.1:53930]
I1201 18:30:25.463080 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (2.9747ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
E1201 18:30:25.465710 1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1201 18:30:25.465953 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
I1201 18:30:25.466586 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (2.8732ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
E1201 18:30:25.467206 1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
I1201 18:30:25.467753 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (7.164101ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
E1201 18:30:25.468083 1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1201 18:30:25.472221 1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1201 18:30:25.472609 1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1201 18:30:25.472730 1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
I1201 18:30:25.472909 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (1.0713ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
I1201 18:30:25.474640 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (1.647ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
E1201 18:30:25.475699 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
I1201 18:30:25.479184 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (20.215901ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
E1201 18:30:25.479482 1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1201 18:30:25.480170 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1201 18:30:25.481352 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1201 18:30:25.482583 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
I1201 18:30:25.483849 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (25.682902ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
E1201 18:30:25.485396 1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1201 18:30:25.486509 1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1201 18:30:25.487726 1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
I1201 18:30:25.488748 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (17.878101ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
I1201 18:30:25.489839 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (18.999301ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
I1201 18:30:25.491108 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (19.478801ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
I1201 18:30:26.476227 1 httplog.go:90] GET /openapi/v2: (1.2015ms) 404 [ 192.168.65.3:46868]
I1201 18:30:28.165256 1 httplog.go:90] GET /healthz: (190.7µs) 200 [kube-probe/1.16+ 10.1.0.1:53966]
I1201 18:30:34.360535 1 httplog.go:90] GET /healthz: (116.3µs) 200 [kube-probe/1.16+ 10.1.0.1:54020]
I1201 18:30:36.350376 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (4.363901ms) 200 [kube-controller-manager/v1.16.6 (linux/amd64) kubernetes/e7f962b/system:serviceaccount:kube-system:resourcequota-controller 192.168.65.3:46868]
I1201 18:30:37.315883 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (2.5138ms) 200 [kube-controller-manager/v1.16.6 (linux/amd64) kubernetes/e7f962b/controller-discovery 192.168.65.3:46868]
I1201 18:30:37.329496 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/%2A/http_server_requests_seconds_count_sum?labelSelector=app%3Dauth-server: (5.072601ms) 404 [kube-controller-manager/v1.16.6 (linux/amd64) kubernetes/e7f962b/system:serviceaccount:kube-system:horizontal-pod-autoscaler 192.168.65.3:46868]
I1201 18:30:38.165664 1 httplog.go:90] GET /healthz: (1.2018ms) 200 [kube-probe/1.16+ 10.1.0.1:54046]
I1201 18:30:44.361064 1 httplog.go:90] GET /healthz: (73.5µs) 200 [kube-probe/1.16+ 10.1.0.1:54108]
I1201 18:30:44.960512 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (1.0876ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
I1201 18:30:44.961265 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (2.1336ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
I1201 18:30:44.961449 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (1.61ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
E1201 18:30:44.961861 1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1201 18:30:44.961923 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1201 18:30:44.962376 1 writers.go:105] apiserver was unable to write a JSON response: http2: stream closed
E1201 18:30:44.963100 1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
E1201 18:30:44.964224 1 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"http2: stream closed"}
E1201 18:30:44.965232 1 writers.go:118] apiserver was unable to write a fallback JSON response: http2: stream closed
I1201 18:30:44.966366 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (5.682901ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
I1201 18:30:44.967578 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1: (8.177501ms) 200 [Go-http-client/2.0 192.168.65.3:46844]
I1201 18:30:48.168761 1 httplog.go:90] GET /healthz: (2.2637ms) 200 [kube-probe/1.16+ 10.1.0.1:54138]
I1201 18:30:48.237637 1 httplog.go:90] GET /apis/custom.metrics.k8s.io/v1beta1?timeout=32s: (1.4318ms) 200 [kube-controller-manager/v1.16.6 (linux/amd64) kubernetes/e7f962b/system:serviceaccount:kube-system:generic-garbage-collector 192.168.65.3:46868]
In configmap prometheus-adapter on kubernetes I have:
- metricsQuery: sum(<<.Series>>{<<.LabelMatchers>>,uri=~"/persons.*"}) by
(<<.GroupBy>>)
name:
as: http_server_requests_seconds_count_sum
matches: ^http_server_requests_seconds_count(.*)
resources:
overrides:
kubernetes_namespace:
resource: namespace
kubernetes_pod_name:
resource: pod
seriesQuery: '{__name__=~"^http_server_requests_seconds_.*"}'
with many diffrent metrics.
When try create Horizonal Pod Autoscaler with my metric I have: unable to get metric ...
In command: kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1" I have many metrics so I think connection is ok.
But my metric isn't there and when I use this command (DOWN), the same error could not find metric
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/http_server_requests_seconds_count_sum"