kubernetes master node and admin user don't have permissions after update

user_12799600 0
Viewed 897

I've googled few days and haven't found any decisions. I've tried to update k8s from 1.19.0 to 1.19.6 In Ubuntu-20. (cluster manually installed k81 - master and k82 - worker node)

# kubeadm upgrade plan
[upgrade/config] Making sure the configuration is correct:
[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[preflight] Running pre-flight checks.
[preflight] Some fatal errors occurred:
    [ERROR CoreDNSUnsupportedPlugins]: couldn't retrieve DNS addon deployments: deployments.apps is forbidden: User "system:node:k81" cannot list resource "deployments" in API group "apps" in the namespace "kube-system"
    [ERROR CoreDNSMigration]: couldn't retrieve DNS addon deployments: deployments.apps is forbidden: User "system:node:k81" cannot list resource "deployments" in API group "apps" in the namespace "kube-system"
    [ERROR kubeDNSTranslation]: configmaps "kube-dns" is forbidden: User "system:node:k81" cannot get resource "configmaps" in API group "" in the namespace "kube-system": no relationship found between node 'k81' and this object
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher

When I try to list roles and permissions under kubernetes-admin user - it shows the same error with permissions:

~# kubectl get rolebindings,clusterrolebindings --all-namespaces
Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "system:node:k81" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope
Error from server (Forbidden): clusterrolebindings.rbac.authorization.k8s.io is forbidden: User "system:node:k81" cannot list resource "clusterrolebindings" in API group "rbac.authorization.k8s.io" at the cluster scope

I can list pods and cluster nodes:

# kubectl get nodes 
NAME   STATUS   ROLES    AGE    VERSION
k81    Ready    master   371d   v1.19.6
k82    Ready    <none>   371d   v1.19.6

# kubectl get pods --all-namespaces 
NAMESPACE                   NAME                                            READY   STATUS      RESTARTS   AGE
gitlab-managed-apps         gitlab-runner-gitlab-runner-6bf497d6c9-g7rhc    1/1     Running     47         27d
gitlab-managed-apps         prometheus-kube-state-metrics-c6bbb8465-8kls5   1/1     Running     3          27d
ingress-nginx               ingress-nginx-controller-848bfcb64d-r6k6k       1/1     Running     3          27d
kube-system                 coredns-f9fd979d6-6dd42                         1/1     Running     1          24h
kube-system                 coredns-f9fd979d6-zjsnz                         1/1     Running     1          24h
kube-system                 csi-nfs-controller-5bd5cb55bc-76xdm             3/3     Running     69         27d
kube-system                 csi-nfs-controller-5bd5cb55bc-mkwmv             3/3     Running     61         27d
kube-system                 csi-nfs-node-b4v4g                              3/3     Running     18         49d
kube-system                 etcd-k81                                        1/1     Running     30         371d
kube-system                 kube-apiserver-k81                              1/1     Running     54         371d
kube-system                 kube-controller-manager-k81                     1/1     Running     27         5d22h
kube-system                 kube-flannel-ds-l4xkx                           1/1     Running     13         371d
kube-system                 kube-flannel-ds-rdm4n                           1/1     Running     5          371d
kube-system                 kube-proxy-4976l                                1/1     Running     5          371d
kube-system                 kube-proxy-g2fn4                                1/1     Running     11         371d
kube-system                 kube-scheduler-k81                              1/1     Running     330        371d
kube-system                 tiller-deploy-f5c865db5-zlgk9                   1/1     Running     5          27d

# kubectl  logs coredns-f9fd979d6-zjsnz  -n kube-system
Error from server (Forbidden): pods "coredns-f9fd979d6-zjsnz" is forbidden: User "system:node:k81" cannot get resource "pods/log" in API group "" in the namespace "kube-system"

# kubectl config get-contexts
CURRENT   NAME                          CLUSTER      AUTHINFO           NAMESPACE
*         kubernetes-admin@kubernetes   kubernetes   kubernetes-admin


# kubectl get csr
No resources found
kubernetes
1 Answers

The solution for the issue is to regenerate the kubeconfig file for the admin:

sudo kubeadm init phase kubeconfig admin --kubeconfig-dir=.

Above command will create the admin.conf file in the current directory (let's say it is /home/user/testing/) so when you are running kubectl commands you need to specify it using --kubeconfig {directory}/admin.conf flag, for example:

sudo kubectl get rolebindings,clusterrolebindings --all-namespaces --kubeconfig /home/user/testing/admin.conf

As you are using /etc/kubernetes/admin.conf file by default, you can delete it and create a new one in /etc/kubernetes directory:

sudo rm /etc/kubernetes/admin.conf
sudo kubeadm init phase kubeconfig admin --kubeconfig-dir=/etc/kubernetes/
user_163919910
Related