GKE metrics-server failing (failed to get delegated authentication kubeconfig)

user_12735064 0
Viewed 212

Google Kubernetes Engine's metrics-server is failing, which is preventing Horizontal Pod Autoscale from working (CPU usage stats are unavailable). When I check on the metrics-server-v0.4.5 deployment, the pod is in CrashLoopBackOff and the logs look like this:

2022-07-08 15:54:01.592 GMT Error: failed to get delegated authentication kubeconfig: failed to get delegated authentication kubeconfig: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied
Error
2022-07-08 15:54:01.593 GMT Usage:
Error
2022-07-08 15:54:01.593 GMT [flags]
Error
2022-07-08 15:54:01.593 GMT
Error
2022-07-08 15:54:01.593 GMT Flags:
Error
2022-07-08 15:54:01.593 GMT --add_dir_header If true, adds the file directory to the header of the log messages
Error
2022-07-08 15:54:01.593 GMT --alsologtostderr log to standard error as well as files
Error
2022-07-08 15:54:01.593 GMT --authentication-kubeconfig string kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenreviews.authentication.k8s.io.
Error
2022-07-08 15:54:01.593 GMT --authentication-skip-lookup If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
Error
2022-07-08 15:54:01.593 GMT --authentication-token-webhook-cache-ttl duration The duration to cache responses from the webhook token authenticator. (default 10s)
Error
2022-07-08 15:54:01.593 GMT --authentication-tolerate-lookup-failure If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.
Error
2022-07-08 15:54:01.593 GMT --bind-address ip The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank or an unspecified address (0.0.0.0 or ::), all interfaces will be used. (default 0.0.0.0)
Error
2022-07-08 15:54:01.593 GMT --cert-dir string The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
Error
2022-07-08 15:54:01.593 GMT --client-ca-file string If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
Error
2022-07-08 15:54:01.593 GMT --contention-profiling Enable lock contention profiling, if profiling is enabled
Error
2022-07-08 15:54:01.593 GMT -h, --help help for this command
Error
2022-07-08 15:54:01.593 GMT --http2-max-streams-per-connection int The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.
Error
2022-07-08 15:54:01.593 GMT --kubeconfig string The path to the kubeconfig used to connect to the Kubernetes API server and the Kubelets (defaults to in-cluster config)
Error
2022-07-08 15:54:01.593 GMT --kubelet-certificate-authority string Path to the CA to use to validate the Kubelet's serving certificates.
Error
2022-07-08 15:54:01.593 GMT --kubelet-client-certificate string Path to a client cert file for TLS.
Error
2022-07-08 15:54:01.593 GMT --kubelet-client-key string Path to a client key file for TLS.
Error
2022-07-08 15:54:01.593 GMT --kubelet-insecure-tls Do not verify CA of serving certificates presented by Kubelets. For testing purposes only.
Error
2022-07-08 15:54:01.593 GMT --kubelet-port int The port to use to connect to Kubelets. (default 10250)
Error
2022-07-08 15:54:01.593 GMT --kubelet-preferred-address-types strings The priority of node address types to use when determining which address to use to connect to a particular node (default [Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP])
Error
2022-07-08 15:54:01.593 GMT --kubelet-use-node-status-port Use the port in the node status. Takes precedence over --kubelet-port flag.
Error
2022-07-08 15:54:01.593 GMT --log-flush-frequency duration Maximum number of seconds between log flushes (default 5s)
Error
2022-07-08 15:54:01.593 GMT --log_backtrace_at traceLocation when logging hits line file:N, emit a stack trace (default :0)
Error
2022-07-08 15:54:01.593 GMT --log_dir string If non-empty, write log files in this directory
Error
2022-07-08 15:54:01.593 GMT --log_file string If non-empty, use this log file
Error
2022-07-08 15:54:01.593 GMT --log_file_max_size uint Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
Error
2022-07-08 15:54:01.593 GMT --logtostderr log to standard error instead of files (default true)
Error
2022-07-08 15:54:01.593 GMT --metric-resolution duration The resolution at which metrics-server will retain metrics. (default 1m0s)
Error
2022-07-08 15:54:01.593 GMT --permit-port-sharing If true, SO_REUSEPORT will be used when binding the port, which allows more than one instance to bind on the same address and port. [default=false]
Error
2022-07-08 15:54:01.593 GMT --profiling Enable profiling via web interface host:port/debug/pprof/ (default true)
Error
2022-07-08 15:54:01.593 GMT --requestheader-allowed-names strings List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
Error
2022-07-08 15:54:01.593 GMT --requestheader-client-ca-file string Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.
Error
2022-07-08 15:54:01.593 GMT --requestheader-extra-headers-prefix strings List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
Error
2022-07-08 15:54:01.593 GMT --requestheader-group-headers strings List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
Error
2022-07-08 15:54:01.593 GMT --requestheader-username-headers strings List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
Error
2022-07-08 15:54:01.593 GMT --secure-port int The port on which to serve HTTPS with authentication and authorization. If 0, don't serve HTTPS at all. (default 443)
Error
2022-07-08 15:54:01.593 GMT --skip_headers If true, avoid header prefixes in the log messages
Error
2022-07-08 15:54:01.593 GMT --skip_log_headers If true, avoid headers when opening log files
Error
2022-07-08 15:54:01.593 GMT --stderrthreshold severity logs at or above this threshold go to stderr (default 2)
Error
2022-07-08 15:54:01.593 GMT --tls-cert-file string File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
Error
2022-07-08 15:54:01.593 GMT --tls-cipher-suites strings Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be used.
Error
2022-07-08 15:54:01.593 GMT Preferred values: TLS_AES_128_GCM_SHA256, TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256, TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256, TLS_RSA_WITH_3DES_EDE_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_RSA_WITH_AES_256_GCM_SHA384.
Error
2022-07-08 15:54:01.593 GMT Insecure values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_ECDSA_WITH_RC4_128_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_ECDHE_RSA_WITH_RC4_128_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_RC4_128_SHA.
Error
2022-07-08 15:54:01.593 GMT --tls-min-version string Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12, VersionTLS13
Error
2022-07-08 15:54:01.593 GMT --tls-private-key-file string File containing the default x509 private key matching --tls-cert-file.
Error
2022-07-08 15:54:01.593 GMT --tls-sni-cert-key namedCertKey A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. The domain patterns also allow IP addresses, but IPs should only be used if the apiserver has visibility to the IP address requested by a client. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
Error
2022-07-08 15:54:01.593 GMT -v, --v Level number for the log level verbosity
Error
2022-07-08 15:54:01.593 GMT --version Show version
Error
2022-07-08 15:54:01.593 GMT --vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Error
2022-07-08 15:54:01.593 GMT
Error
2022-07-08 15:54:01.688 GMT panic: failed to get delegated authentication kubeconfig: failed to get delegated authentication kubeconfig: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied goroutine 1 [running]: main.main() /go/src/sigs.k8s.io/metrics-server/cmd/metrics-server/metrics-server.go:39 +0xfc

From the logs it looks like there is some kind of permissions problem, but metrics-server is a default service from GKE, not something that I've configured, so I'm confused why it doesn't have permissions and not sure how to go about fixing the problem.

Any help would be appreciated.

google-kubernetes-engine google-cloud-platform
1 Answers

This issue could be due to the node pool version that is running in your GKE cluster.

Please be aware that the nodepool version of the nodepool/s “nodepool_name” is more than two minor versions older than the control plane version. Kubernetes version and version skew support policy only guarantees that control planes are compatible with nodes up to two minor versions older; clusters where the control plane and any nodes are farther apart may experience issues.

This version skew can happen due to having auto-upgrade disabled on the nodepool level as the control plane gets upgraded periodically.

To resolve the version skew, you need to upgrade the node pools that are running the incompatible version of Kubernetes. To do so, you can follow this guide.

If you are concerned about a disruption on the workloads running on the nodepool, please take a look at the “Migrating workloads to different machine types” tutorial. This method should allow you to create a nodepool with the proper version and migrate the workloads, minimizing downtime for your application.

user_178941040
Related