Issues after running kubeadm reset

user_12990614 0
Viewed 824

I was having issues with kubeadm init, and so i ran kubeadm reset and then kubeadm init and the problem at hand went away, but now I have another problem and that is that when I run kubectl get all, I get the following response:

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 3h6m
Error from server (Forbidden): replicationcontrollers is forbidden: User "system:node:abc-server.localdomain" cannot list resource "replicationcontrollers" in API group "" in the namespace "default"
Error from server (Forbidden): daemonsets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "daemonsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): deployments.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "deployments" in API group "apps" in the namespace "default"
Error from server (Forbidden): replicasets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "replicasets" in API group "apps" in the namespace "default"
Error from server (Forbidden): statefulsets.apps is forbidden: User "system:node:abc-server.localdomain" cannot list resource "statefulsets" in API group "apps" in the namespace "default"
Error from server (Forbidden): horizontalpodautoscalers.autoscaling is forbidden: User "system:node:abc-server.localdomain" cannot list resource "horizontalpodautoscalers" in API group "autoscaling" in the namespace "default"
Error from server (Forbidden): jobs.batch is forbidden: User "system:node:abc-server.localdomain" cannot list resource "jobs" in API group "batch" in the namespace "default"
Error from server (Forbidden): cronjobs.batch is forbidden: User "system:node:abc-server.localdomain" cannot list resource "cronjobs" in API group "batch" in the namespace "default"

I've exhausted my googling abilities with my limited kubernetes vocabulary, so hoping someone here could help me with the following:

  1. what's happening?! (is this a RBAC Authorization issue?)
  2. how can i resolve this? as this is a dev environment that will definitely require some clean up, I don't mind a quick and dirty way just so i can continue with the task at hand (which is to just get things up and running again)
kubectl kubernetes
2 Answers

As @Software Engineer mentioned in his comment there is a github issue with a fix for that:

User neolit123 on github posted this solution:

getting a permission error during pod network setup, means you are trying to kubectl apply manifest files using a kubeconfig file which does not have the correct permissions.

make sure that your /etc/kubernetes/admin.conf is generated by kubeadm and contains kubernetes-admin as the user.

root@master:~# kubectl auth can-i create deploy

which kubeconfig is this command using?
try

root@master:~# KUBECONFIG=/etc/kubernetes/admin.conf kubectl auth can-i create deploy

I wanted to check the release notes, but there is no much information, or I don't know interpret it. Does anyone have any information about what are the changes, or what am I doing wrong?

AFAIK, there is no such change that breaks this between 1.14.4 and .3.

user_120144340

Answers to the best of my ability without reproducing the error:

Yes, this looks like a RBAC issue. You ran:

kubectl get all

As you might know, kubectl handles and locates the Kubernetes API server (also known as kube-apiserver in the cluster). Because the API server returns 403 (forbidden), we can proceed knowing that this is an authorization issue.

That is, we can request resources from the kube-apiserver (authentication) but we don't have the privileges to: User "system:node:abc-server.localdomain" does not have the permission to perform a HTTP GET request on such and so objects. Moreover, it's likely that this user doesn't have permission to perform HTTP requests (CRUD -- Create Read Update Delete) for any objects in the Kubernetes cluster.

In RBAC terms, there exists a Role that defines permissions and a RoleBinding that couples the user with that Role to get the permissions. As an example, a cluster has a Role that is equal to Unix/Linux root in terms of permissions, which is bound to the kube-apiserver. Obviously, we're not gonna use that role but generally, we can use those permissions to create roles that have CRUD permissions.

So when you bootstrapped the control plane, system:node:abc-server.localdomain should have been bound to a Role that gives permissions to perform a Read request on the objects in the default namespace. But I'm guessing that didn't happen.

I'm just gonna say and it would be most easy to restart the process by wiping the hosts and recreating the cluster. If you run into errors once more, please document all the steps you took so anyone can reproduce the problem and apply more thorough troubleshooting.

user_83578260
Related