tcp-established in /etc/criu/default.conf not working with docker checkpoint create?

[pietschy]

Hi there,

I've created the file /etc/criu/default.conf and put a single line with tcp-established in it but am still getting the following error when I try and checkpoint my container.

docker checkpoint create 38c5d6de19c6 test_checkpoint
Error response from daemon: Cannot checkpoint container 38c5d6de19c6: runc did not terminate sucessfully: criu failed: type NOTIFY errno 0 path= /var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/38c5d6de19c6bfde5bf6053796b75bc2adbaafcfa154cce6d46284ab62058760/criu-dump.log: unknown

And the log file contains the line

(00.183817) sockets: 	Searching for socket 10792 (family 1.0)
(00.183819) unix: Dumping semi-closed connection
(00.183825) sockets: No filter for socket
(00.183826) unix: Dumping unix socket at 243
(00.183827) unix: 	Dumping: ino 67474 peer_ino 0 family    1 type    1 state  1 name (null)
(00.183831) unix: 	Dumped: id 0x19b ino 67474 peer 0 type 1 state 1 name 0 bytes
(00.183837) 6047 fdinfo 244: pos:                0 flags:                2/0
(00.183841) sockets: 	Searching for socket 10797 (family 2.6)
(00.183842) Error (criu/sk-inet.c:188): inet: Connected TCP socket, consider using --tcp-established option.
(00.183911) ----------------------------------------
(00.183915) Error (criu/cr-dump.c:1347): Dump files (pid: 6047) failed with -1

I tested /etc/criu/default.conf by adding an invalid option and and running criu check. It barfed with an appropriate message as did docker checkpoint.

Any ideas what I'm doing wrong?

Thanks & cheers

Version info:
Fedora release 29 (Twenty Nine)
Kernel: 4.18.16-300.fc29.x86_64
Criu: 3.11-2.fc30
Docker: 18.09.1, build 4c52b90

[adrianreber]

The version of runc you have it probably too old. You need at least a version which includes opencontainers/runc#1933.

You could try to open a bug against Fedora's runc in bugzilla.redhat.com. runc usually gets updated pretty quickly in Fedora.

You could also try Podman as it has support to specify --tcp-established on the command-line: containers/podman#1848

[rst0git]

I tested /etc/criu/default.conf by adding an invalid option and and running criu check. It barfed with an appropriate message as did docker checkpoint.
Any ideas what I'm doing wrong?

@pietschy When adding options in /etc/criu/default.conf they will be overwritten by the RPC options provided by runc (docker), and --tcp-established is set to false by default. You need to add your options in /etc/criu/runc.conf. The options defined in this file will overwrite the ones specified by runc (docker).

@adrianreber This looks like a very common question. Maybe we should explain it better in https://criu.org/Configuration_files, and perhaps mention about runc.conf and podman. WDYT?

[pietschy]

Thanks guys,

Looks like I'm running the latest runc 1.0.0-rc6+dev release so at this stage I'll wait for the next one and try again then (I'm guessing Fedora won't consider releasing a new package until that happens anyway).

As for the docs, Google took me to https://criu.org/Docker first so having it referenced there would have helped a lot.

[adrianreber]

Looking at https://src.fedoraproject.org/rpms/runc/commits/master I see that the changes are already in Fedora's development branch and my experience with the runc package is that it updated pretty fast if you ask. So I really recommend to open a ticket and ask. Or try rawhide runc from here: https://koji.fedoraproject.org/koji/buildinfo?buildID=1180067

Then, @rst0git was right, you need to use the configuration file /etc/criu/runc.conf once you have that latest version of runc. Also, please try out Podman to see if it works with --tcp-established.

[adrianreber]

@adrianreber This looks like a very common question. Maybe we should explain it better in https://criu.org/Configuration_files, and perhaps mention about runc.conf and podman. WDYT?

@rst0git sure, it's a wiki 😀, please just change it

[pietschy]

Thanks all for your help, I've created a bug with fedora.

https://bugzilla.redhat.com/show_bug.cgi?id=1670223

[avagin]

The fedora bug was closed. Should we close this one?

[pietschy]

I can confirm checkpointing is working for my containers in the latest fedora release so I'm happy to close this. Thanks for all your help guys.

[kohanyirobert]

@adrianreber This looks like a very common question. Maybe we should explain it better in https://criu.org/Configuration_files, and perhaps mention about runc.conf and podman. WDYT?

I second it. I've spent a few hours trying to make this work when finally I've found this comment.