One of our VMs was power cycled, and now docker won't startup.
When I try to start the service via systemctl start docker-standalone, I get:
Job for docker-standalone.service failed because the control process exited with error code. See "systemctl status docker-standalone.service" and "journalctl -xe" for details.
When I check journalctl -xe, I see this:
Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.689962814-06:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit sta Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.689984624-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690001784-06:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured" Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690010545-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690029639-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690106664-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690248278-06:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker-standalone/containerd/daemon/io.c Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690261292-06:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690274990-06:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured" Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690282381-06:00" level=info msg="metadata content store policy set" policy=shared Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690356752-06:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690370466-06:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690399678-06:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690427410-06:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690438870-06:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690449099-06:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690462529-06:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690476538-06:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690489694-06:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690511122-06:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690524355-06:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1 Oct 29 13:56:16 server35.mydomain.com dockerd[4382]: Failed to start containerd: timeout waiting for containerd to start Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service: main process exited, code=exited, status=1/FAILURE Oct 29 13:56:16 server35.mydomain.com systemd[1]: Failed to start Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has failed.
--
-- The result is failed. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Unit docker-standalone.service entered failed state. Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service failed. Oct 29 13:56:16 server35.mydomain.com polkitd[800]: Unregistered Authentication Agent for unix-process:4406:22575 (system bus name :1.24, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected f Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service holdoff time over, scheduling restart. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Stopped Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has finished shutting down. Oct 29 13:56:16 server35.mydomain.com systemd[1]: start request repeated too quickly for docker-standalone.service Oct 29 13:56:16 server35.mydomain.com systemd[1]: Failed to start Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has failed.
--
-- The result is failed. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Unit docker-standalone.service entered failed state. Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service failed.Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.689962814-06:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit sta Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.689984624-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.devmapper\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690001784-06:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured" Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690010545-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.native\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690029639-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.overlayfs\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690106664-06:00" level=info msg="loading plugin \"io.containerd.snapshotter.v1.zfs\"..." type=io.containerd.snapshotter.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690248278-06:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/docker-standalone/containerd/daemon/io.c Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690261292-06:00" level=info msg="loading plugin \"io.containerd.metadata.v1.bolt\"..." type=io.containerd.metadata.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690274990-06:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured" Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690282381-06:00" level=info msg="metadata content store policy set" policy=shared Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690356752-06:00" level=info msg="loading plugin \"io.containerd.differ.v1.walking\"..." type=io.containerd.differ.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690370466-06:00" level=info msg="loading plugin \"io.containerd.gc.v1.scheduler\"..." type=io.containerd.gc.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690399678-06:00" level=info msg="loading plugin \"io.containerd.service.v1.introspection-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690427410-06:00" level=info msg="loading plugin \"io.containerd.service.v1.containers-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690438870-06:00" level=info msg="loading plugin \"io.containerd.service.v1.content-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690449099-06:00" level=info msg="loading plugin \"io.containerd.service.v1.diff-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690462529-06:00" level=info msg="loading plugin \"io.containerd.service.v1.images-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690476538-06:00" level=info msg="loading plugin \"io.containerd.service.v1.leases-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690489694-06:00" level=info msg="loading plugin \"io.containerd.service.v1.namespaces-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690511122-06:00" level=info msg="loading plugin \"io.containerd.service.v1.snapshots-service\"..." type=io.containerd.service.v1 Oct 29 13:56:11 server35.mydomain.com dockerd[4382]: time="2021-10-29T13:56:11.690524355-06:00" level=info msg="loading plugin \"io.containerd.runtime.v1.linux\"..." type=io.containerd.runtime.v1 Oct 29 13:56:16 server35.mydomain.com dockerd[4382]: Failed to start containerd: timeout waiting for containerd to start Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service: main process exited, code=exited, status=1/FAILURE Oct 29 13:56:16 server35.mydomain.com systemd[1]: Failed to start Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has failed.
--
-- The result is failed. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Unit docker-standalone.service entered failed state. Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service failed. Oct 29 13:56:16 server35.mydomain.com polkitd[800]: Unregistered Authentication Agent for unix-process:4406:22575 (system bus name :1.24, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected f Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service holdoff time over, scheduling restart. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Stopped Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has finished shutting down. Oct 29 13:56:16 server35.mydomain.com systemd[1]: start request repeated too quickly for docker-standalone.service Oct 29 13:56:16 server35.mydomain.com systemd[1]: Failed to start Docker Application Container Engine Standalone Version.
-- Subject: Unit docker-standalone.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker-standalone.service has failed.
--
-- The result is failed. Oct 29 13:56:16 server35.mydomain.com systemd[1]: Unit docker-standalone.service entered failed state. Oct 29 13:56:16 server35.mydomain.com systemd[1]: docker-standalone.service failed.
I noticed this key line:
Failed to start containerd: timeout waiting for containerd to start
We tried reinstalling docker, but that didn't help.
Our versions are:
$ ds version
Client: Docker Engine - Community
Version: 20.10.8
API version: 1.40
Go version: go1.16.6
Git commit: 3967b7d
Built: Fri Jul 30 19:55:49 2021
OS/Arch: linux/amd64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 19.03.13
API version: 1.40 (minimum version 1.12)
Go version: go1.13.15
Git commit: 4484c46d9d
Built: Wed Sep 16 17:02:21 2020
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.4.9
GitCommit: e25210fe30a0a703442421b0f60afac609f950a3
runc:
Version: 1.0.1
GitCommit: v1.0.1-0-g4144b63
docker-init:
Version: 0.18.0
GitCommit: fec3683
We're running Centos 7 with the Linux kernel 3.10.0-514.el7.x86_64
We also tried removing the docker files in /var/run based on the suggestion in this SO post, but that was Ubunutu and involved snap, and clearing out the docker files in /var/run didn't help.
To eliminate any confusion about the service name, ds is just a helper script defined as:
#!/bin/bash
docker -H unix:///var/run/docker-standalone.sock "$@"
How do we get docker to start again?