Kubernetes RBAC unable to upgrade connection: Forbidden (user=system:anonymous, verb=create, resource=nodes, subresource=proxy)

user_3745378 0
Viewed 12224

I'm running Kubernetes 1.6.2 with RBAC enabled. I've created a user kube-admin that has the following Cluster Role binding

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: k8s-admin
subjects:
- kind: User
  name: kube-admin
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: cluster-admin
  apiGroup: rbac.authorization.k8s.io

When I attempt to kubectl exec into a running pod I get the following error.

kubectl -n kube-system exec -it kubernetes-dashboard-2396447444-1t9jk -- /bin/bash
error: unable to upgrade connection: Forbidden (user=system:anonymous, verb=create, resource=nodes, subresource=proxy)

My guess is I'm missing a ClusterRoleBinding ref, which role am I missing?

kubectl kubernetes
2 Answers

I had this exact same error happening, but for me the problem was due to my setup with kops. I wanted to share my result here because it may help someone in the future.

With a bug that exists in kops v1.19.1, the config for kubelet anonymousAuth must be explicitly set to false

I was using kops version 1.19, and upgrading my cluster from k8s v1.11 to v1.19. After the upgrade is when I started seeing this error when trying to run kubectl port-forward, kubectl logs, kubectl exec, helm list, etc. The issue was a combination of a bug in kops and not having the anonymousAuth configuration set, or having it set to true. With this bug in kops, the config for kubelet anonymousAuth must be set explicitly to false.

To Fix

Edit the cluster

$ kops edit cluster

Add the config under spec.kubelet.anonymousAuth, i.e.

spec:
  kubelet:
    anonymousAuth: false

Update the cluster

$ kops update cluster --yes

$ kops rolling-update cluster --yes

Related

PR that fixes the immediate issue

PR that fixes a related issue

Related kops docs

user_47466480
Related