Kubernetes was working fine till today when something happened and it stopped responding. Kubernetes itself is working and I can access it in my browser, but I can't do it in VM terminal:
root@master01:/home/dev-admin# kubectl get pods --all-namespaces
The connection to the server 192.168.32.10:6443 was refused - did you specify the right host or port?
I checked config files and everything is normal (hosts, usernames, etc.), but when I checked docker I got following response:
root@master01:/home/dev-admin# systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/etc/systemd/system/docker.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/docker.service.d
└─docker-dns.conf, docker-options.conf
Active: failed (Result: exit-code) since Thu 2022-07-21 09:41:47 +06; 13min ago
TriggeredBy: ● docker.socket
Docs: http://docs.docker.com
Process: 18044 ExecStart=/usr/bin/dockerd $DOCKER_OPTS $DOCKER_STORAGE_OPTIONS $DOCKER_DNS_OPTIONS (cod>
Main PID: 18044 (code=exited, status=1/FAILURE)
Jul 21 09:41:47 master01 systemd[1]: Failed to start Docker Application Container Engine.
Jul 21 09:42:02 master01 systemd[1]: docker.service: Start request repeated too quickly.
Jul 21 09:42:02 master01 systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 21 09:42:02 master01 systemd[1]: Failed to start Docker Application Container Engine.
Jul 21 09:42:02 master01 systemd[1]: docker.service: Start request repeated too quickly.
Jul 21 09:42:02 master01 systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 21 09:42:02 master01 systemd[1]: Failed to start Docker Application Container Engine.
Jul 21 09:42:03 master01 systemd[1]: docker.service: Start request repeated too quickly.
Jul 21 09:42:03 master01 systemd[1]: docker.service: Failed with result 'exit-code'.
Jul 21 09:42:03 master01 systemd[1]: Failed to start Docker Application Container Engine.
Starting it failed too:
root@master01:/home/dev-admin# systemctl start docker
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
Output of journalctl -xe:
Jul 21 10:11:50 master01 systemd[1]: Starting Docker Socket for the API.
-- Subject: A start job for unit docker.socket has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit docker.socket has begun execution.
--
-- The job identifier is 133321.
Jul 21 10:11:50 master01 systemd[1]: Listening on Docker Socket for the API.
-- Subject: A start job for unit docker.socket has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit docker.socket has finished successfully.
--
-- The job identifier is 133321.
Jul 21 10:11:50 master01 systemd[1]: docker.service: Start request repeated too quickly.
Jul 21 10:11:50 master01 systemd[1]: docker.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit docker.service has entered the 'failed' state with result 'exit-code'.
Jul 21 10:11:50 master01 systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: A start job for unit docker.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit docker.service has finished with a failure.
--
-- The job identifier is 133248 and the job result is failed.
Jul 21 10:11:50 master01 systemd[1]: docker.socket: Failed with result 'service-start-limit-hit'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit docker.socket has entered the 'failed' state with result 'service-start-limit-hit'.
Since everything else is working, I'm guessing that kubernetes can't connect because of Docker. I tried every solution I could find but it still doesn't work. Reinstalling it is not and option (or very last option), so if anyone experienced same thing or knows how to solve it please let me know.
EDIT: Forgot to add debugging.
After I run sudo dockerd --debug it starts debugging, but gets stuck at Calling GET:
DEBU[2022-07-21T11:28:03.502069335+06:00] Calling HEAD /_ping
DEBU[2022-07-21T11:28:03.502440946+06:00] Calling GET /v1.40/info
DEBU[2022-07-21T11:28:03.510771199+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:03.523425727+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:03.530174695+06:00] Calling GET /v1.40/info
DEBU[2022-07-21T11:28:03.536275178+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:03.542823963+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:03.551757708+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:03.560728462+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:03.568877107+06:00] Calling HEAD /_ping
DEBU[2022-07-21T11:28:03.569144592+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:03.610711851+06:00] Calling GET /v1.40/info
DEBU[2022-07-21T11:28:13.746935574+06:00] Calling HEAD /_ping
DEBU[2022-07-21T11:28:13.747328082+06:00] Calling GET /v1.40/info
DEBU[2022-07-21T11:28:13.756079740+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:13.764205989+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:13.772453878+06:00] Calling GET /v1.40/info
DEBU[2022-07-21T11:28:13.779109018+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:13.784794616+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:13.797833614+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:13.807122074+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:13.813391070+06:00] Calling HEAD /_ping
DEBU[2022-07-21T11:28:13.813706861+06:00] Calling GET /v1.40/version
DEBU[2022-07-21T11:28:13.852937729+06:00] Calling GET /v1.40/info