kubectl exec error dialing backend: x509: certificate signed by unknown authority

user_5145890 0
Viewed 2889

After a long struggle I just created my cluster, deployed a sample container busybox now i am trying to run the command exec and i get the following error:

error dialing backend: x509: certificate signed by unknown authority

How do i solve this one: here is the command output with v=9 log level. kubectl exec -v=9 -ti busybox -- nslookup kubernetes I also noticed in the logs that this curl command that failed is actually the second command the first GET command passed and it return results without any issues.. ( GET https://myloadbalancer.local:6443/api/v1/namespaces/default/pods/busybox 200 OK)

curl -k -v -XPOST  -H "X-Stream-Protocol-Version: v4.channel.k8s.io" -H "X-Stream-Protocol-Version: v3.channel.k8s.io" -H "X-Stream-Protocol-Version: v2.channel.k8s.io" -H "X-Stream-Protocol-Version: channel.k8s.io" -H "User-Agent: kubectl/v1.19.0 (linux/amd64) kubernetes/e199641" 'https://myloadbalancer.local:6443/api/v1/namespaces/default/pods/busybox/exec?command=nslookup&command=kubernetes&container=busybox&stdin=true&stdout=true&tty=true'
I1018 02:19:40.776134  129813 round_trippers.go:443] POST https://myloadbalancer.local:6443/api/v1/namespaces/default/pods/busybox/exec?command=nslookup&command=kubernetes&container=busybox&stdin=true&stdout=true&tty=true 500 Internal Server Error in 43 milliseconds
I1018 02:19:40.776189  129813 round_trippers.go:449] Response Headers:
I1018 02:19:40.776206  129813 round_trippers.go:452]     Content-Type: application/json
I1018 02:19:40.776234  129813 round_trippers.go:452]     Date: Sun, 18 Oct 2020 02:19:40 GMT
I1018 02:19:40.776264  129813 round_trippers.go:452]     Content-Length: 161
I1018 02:19:40.776277  129813 round_trippers.go:452]     Cache-Control: no-cache, private
I1018 02:19:40.777904  129813 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "error dialing backend: x509: certificate signed by unknown authority",
  "code": 500
}]
F1018 02:19:40.778081  129813 helpers.go:115] Error from server: error dialing backend: x509: certificate signed by unknown authority
goroutine 1 [running]:

Adding more information: This is on UBUNTU 20.04. I went through step by step creating my cluster manually as a beginner I need that experience instead of spinning up with tools like kubeadm or minikube

xxxx@master01:~$ kubectl exec -ti busybox -- nslookup kubernetes
Error from server: error dialing backend: x509: certificate signed by unknown authority
xxxx@master01:~$ kubectl get pods --all-namespaces
NAMESPACE              NAME                                         READY   STATUS    RESTARTS   AGE
default                busybox                                      1/1     Running   52         2d5h
kube-system            coredns-78cb77577b-lbp87                     1/1     Running   0          2d5h
kube-system            coredns-78cb77577b-n7rvg                     1/1     Running   0          2d5h
kube-system            weave-net-d9jb6                              2/2     Running   7          2d5h
kube-system            weave-net-nsqss                              2/2     Running   0          2d14h
kube-system            weave-net-wnbq7                              2/2     Running   7          2d5h
kube-system            weave-net-zfsmn                              2/2     Running   0          2d14h
kubernetes-dashboard   dashboard-metrics-scraper-7b59f7d4df-dhcpn   1/1     Running   0          2d3h
kubernetes-dashboard   kubernetes-dashboard-665f4c5ff-6qnzp         1/1     Running   7          2d3h
tinashe@master01:~$ kubectl logs busybox
Error from server: Get "https://worker01:10250/containerLogs/default/busybox/busybox": x509: certificate signed by unknown authority
xxxx@master01:~$

xxxx@master01:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:41:49Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}
kubectl kubelet kubernetes
1 Answers

**hijacking my own comment to indicate that while the below did help for a minute it didn't actually solve my problem, and I discovered a much better solution here (leaving below in place). my cluster operator kube-apiserver was degraded, causing my certificate failures. Resolving that degradation was necessary to resolve the overarching problem, resulting in x509 errors.

https://access.redhat.com/solutions/4849711

Had a similar issue with my OCP4 cluster. Turned out the issue was that the nodes provided their own self-signed cert when they spun up and missed getting added to the trusted connections, failing my queries.

Resolved by pulling the cert using openssl tools (example below) , then adding that resulting pem file to /etc/ssl/certs (allowed it to recognize the server and resolve the failure).

Of course the proper way to fix this is to assign proper certificates, but this here ought to get you rolling with what you've started with. Posting in case it helps others in the future.

determine the internal IP of the node that's faulting:

$ oc get nodes #(kubectl get nodes)

master01

$ oc describe node master01 #(kubectl describe node master01)

...output omitted... internal IP: 192.168.50.10

pull the cert information from the self-signed certificate on faulty-node:

$ echo | openssl s_client -showcert
s -servername 192.168.50.10 -connect 192.168.50.10:443 2>/
dev/null | openssl x509 -inform pem -noout -text > nodename.pem

copy the resulting pem file to your local certs file.

$ cp nodename.pem /etc/ssl/certs

try your request again.

user_163173590
Related