2 identical Kubernetes Cluster Failed 48 hours apart

liz french 0
Viewed 84

Error message in the logs on the slave nodes reads, this is from the kube-proxy.log:

*Failed to watch v1beta1.EndpointSlice: the server has asked for the client to provide credentials.

Clusters were generated via automated KOPS utility. Both clusters have been running fine for 18 months on EC2 instances on AWS. Does anyone have any pointers as to what this might be?

Here is the relevant snippet from the log:

I0913 02:36:28.918119       1 proxier.go:826] syncProxyRules took 120.360802ms
I0913 02:36:29.918644       1 proxier.go:871] Syncing iptables rules
I0913 02:36:29.989692       1 proxier.go:826] syncProxyRules took 71.416518ms
E0913 02:37:24.345615       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: the server has asked for the client to provide credentials (get endpointslices.discovery.k8s.io)
E0913 02:37:25.260072       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: Unauthorized
E0913 02:37:27.280699       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: Unauthorized
E0913 02:37:30.846798       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: Unauthorized
E0913 02:37:39.239017       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: Unauthorized
E0913 02:37:57.653121       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1beta1.EndpointSlice: failed to list *v1beta1.EndpointSlice: Unauthorized
E0913 02:38:04.190383       1 reflector.go:138] k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: the server has asked for the client to provide credentials (get services)
E
kops kubernetes
1 Answers

The node certificates are only valid for ~18 months +/- random skew. After than, most core components of K8s will fail. By the looks of it, your nodes are older than that. The solution is to rotate your nodes.

This should rarely happen as both kOps and K8s both only support 3 versions going back, meaning your cluster should have been updated about once a year minimum.

Ole Markus With0
Related