Unable to check ETCD cluster health

user_7977874 0
Viewed 5174

After following official guidance from (https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/setup-ha-etcd-with-kubeadm/) for HA external ETCD, I was trying to check etcd cluster health status, but, it threw some error which I'm struggling to solve. Please help.

Command used to check etcd cluster health that threw an error:

docker run --rm -it \
--net host \
-v /etc/kubernetes:/etc/kubernetes k8s.gcr.io/etcd:3.4.3-0 etcdctl \
--cert /etc/kubernetes/pki/etcd/peer.crt \
--key /etc/kubernetes/pki/etcd/peer.key \
--cacert /etc/kubernetes/pki/etcd/ca.crt \
--endpoints https://${HOST0}:2379 endpoint health --cluster

Three bare-metal hosts, all ufw enabled, 2379 and 2380 ports allowed on entire hosts.

Error:

Error: failed to fetch endpoints from etcd cluster member list: context deadline exceeded ```

**etcd docker image**:
``` k8s.gcr.io/etcd     3.4.3-0             303ce5db0e90        9 months ago        288MB ````
etcd kubernetes high-availability
1 Answers

Context deadline exceeded is an unclear error returned by grpc client when it can't establish the connection. you can set ETCDCTL_API=2, then you can get the right error message.
Also, you can change some code in etcd to debug this error.
see #10087

You can solve this issue when I applied the right cert/key pair.

Assuming you're using kubeadm to spin up the cluster, there should be a couple of cert/key pairs under the folder:

# ls -l /etc/kubernetes/pki/etcd/
total 32
-rw-r--r--    1 root     root          1017 Nov 12 15:32 ca.crt
-rw-------    1 root     root          1679 Nov 12 15:32 ca.key
-rw-r--r--    1 root     root          1094 Nov 12 15:32 healthcheck-client.crt
-rw-------    1 root     root          1675 Nov 12 15:32 healthcheck-client.key
-rw-r--r--    1 root     root          1180 Nov 12 15:32 peer.crt
-rw-------    1 root     root          1675 Nov 12 15:32 peer.key
-rw-r--r--    1 root     root          1180 Nov 12 15:32 server.crt
-rw-------    1 root     root          1679 Nov 12 15:32 server.key

# etcdctl --version
etcdctl version: 3.3.1
API version: 2

# ETCDCTL_API=3 etcdctl snapshot save snapshot.db \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key
Snapshot saved at snapshot.db

# ETCDCTL_API=3 etcdctl --write-out=table snapshot status snapshot.db
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| b9d500f7 |    72966 |       1194 |     4.9 MB |

You can also try to follow this instruction error-context-deadline-exceeded-accessing.

Take a look: context-deadline-exceeded.

user_113003820
Related