Failed to restart docker after weave stop

[zhouhaibing089]

The way that I start weave:

# mkdir -p /opt/cni/bin
# mkdir -p /etc/cni/net.d
# weave setup
# weave launch ${peers}
# weave expose

Then I decide to stop weave and restart docker:

# weave stop
# systemctl restart docker

And docker failed to restart, the logs shows:

Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.735859365Z" level=warning msg="open /sys/fs/cgroup/freezer/system.slice/docker-0e1e2d76cc3eae2c800a5826496b8d3c79118e52ce57904316ec97695603457f.scope/freezer.state: no such file or directory"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.736145855Z" level=warning msg="open /sys/fs/cgroup/freezer/system.slice/docker-0e1e2d76cc3eae2c800a5826496b8d3c79118e52ce57904316ec97695603457f.scope/freezer.state: no such file or directory"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.818121458Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/0e1e2d76cc3eae2c800a5826496b8d3c79118e52ce57904316ec97695603457f/shm: invalid argument\nfailed to umount /var/lib/docker/containers/0e1e2d76cc3eae2c800a5826496b8d3c79118e52ce57904316ec97695603457f/mqueue: invalid argument"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.818206465Z" level=error msg="Error unmounting device 0e1e2d76cc3eae2c800a5826496b8d3c79118e52ce57904316ec97695603457f: UnmountDevice: device not-mounted id 0e1e2d76cc3eae2c800a5826496b8d3c79118e52ce57904316ec97695603457f"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.818622698Z" level=error msg="Failed to start container 0e1e2d76cc3eae2c800a5826496b8d3c79118e52ce57904316ec97695603457f: [8] System error: not a directory"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.836990680Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/060f1aa5d814916ad52e5416f06dfa7ea438d01efe7cdf71e36e8dde17ebcd4e/shm: invalid argument\nfailed to umount /var/lib/docker/containers/060f1aa5d814916ad52e5416f06dfa7ea438d01efe7cdf71e36e8dde17ebcd4e/mqueue: invalid argument"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.837107999Z" level=error msg="Error unmounting device 060f1aa5d814916ad52e5416f06dfa7ea438d01efe7cdf71e36e8dde17ebcd4e: UnmountDevice: device not-mounted id 060f1aa5d814916ad52e5416f06dfa7ea438d01efe7cdf71e36e8dde17ebcd4e"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.837239364Z" level=error msg="Failed to start container 060f1aa5d814916ad52e5416f06dfa7ea438d01efe7cdf71e36e8dde17ebcd4e: [8] System error: not a directory"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Timestamp: 2016-06-13 00:57:47.856721414 +0000 UTC
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Code: System error
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]:
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Message: not a directory
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]:
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Frames:
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: ---
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: 0: setupRootfs
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Package: github.com/opencontainers/runc/libcontainer
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: File: rootfs_linux.go@40
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: ---
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: 1: Init
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: File: standard_init_linux.go@57
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: ---
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: 2: StartInitialization
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: File: factory_linux.go@242
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: ---
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: 3: initializer
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Package: github.com/docker/docker/daemon/execdriver/native
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: File: init.go@35
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: ---
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: 4: Init
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Package: github.com/docker/docker/pkg/reexec
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: File: reexec.go@26
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: ---
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: 5: main
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Package: main
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: File: docker.go@18
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: ---
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: 6: main
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Package: runtime
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: File: proc.go@111
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: ---
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: 7: goexit
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: Package: runtime
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com docker[25704]: File: asm_amd64.s@1721
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.896777012Z" level=warning msg="exit status 1"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.984058037Z" level=warning msg="failed to cleanup ipc mounts:\nfailed to umount /var/lib/docker/containers/700e852df8803b5a2e818f3e04152c44c7f49ecb8caf2d7620746faefd2e1975/shm: invalid argument\nfailed to umount /var/lib/docker/containers/700e852df8803b5a2e818f3e04152c44c7f49ecb8caf2d7620746faefd2e1975/mqueue: invalid argument"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.984127273Z" level=error msg="Error unmounting device 700e852df8803b5a2e818f3e04152c44c7f49ecb8caf2d7620746faefd2e1975: UnmountDevice: device not-mounted id 700e852df8803b5a2e818f3e04152c44c7f49ecb8caf2d7620746faefd2e1975"
Jun 13 00:57:47 node-1-5168.phx01.dev.example.com forward-journal[25705]: time="2016-06-13T00:57:47.984274003Z" level=error msg="Failed to start container 700e852df8803b5a2e818f3e04152c44c7f49ecb8caf2d7620746faefd2e1975: [8] System error: not a directory"

[zhouhaibing089]

docker version

# docker version
Client:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-9.gitee06d03.fc23.x86_64
 Go version:      go1.5.4
 Git commit:      ee06d03/1.9.1
 Built:
 OS/Arch:         linux/amd64

Server:
 Version:         1.9.1
 API version:     1.21
 Package version: docker-1.9.1-9.gitee06d03.fc23.x86_64
 Go version:      go1.5.4
 Git commit:      ee06d03/1.9.1
 Built:
 OS/Arch:         linux/amd64

weave version

# weave version
weave script 1.5.2
weave router 1.5.2
weave proxy  1.5.2
weave plugin 1.5.2

[zhouhaibing089]

Before restarting the docker:

# docker ps
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS               NAMES
060f1aa5d814        weaveworks/plugin:1.5.2      "/home/weave/plugin"     15 hours ago        Up 15 hours                             weaveplugin
0e1e2d76cc3e        weaveworks/weaveexec:1.5.2   "/home/weave/weavepro"   15 hours ago        Up 15 hours                             weaveproxy
700e852df880        weaveworks/weave:1.5.2       "/home/weave/weaver -"   15 hours ago        Up 15 hours                             weave

And the container id showed in the logs are all relevant with these containers.

[bboreham]

Hi @zhouhaibing089, is it possible for you to upgrade Docker? They fixed a number of restart issues after 1.9.1, e.g. moby/libnetwork#813

Also moby/moby#18076 may relate to the mqueue error message in your logs, again fixed after 1.9.1

[rade]

I am going to assume that @bboreham's assessment is right and this is a docker issue. If not, please re-open.