Runc did not terminate successfully: criu failed: type NOTIFY errno 0 #1118
Comments
|
Can you show us the output of |
|
I'm not sure why they are different. I am on 8.1 for both RHEL and CentOS currently. I'm getting an error when I try to attach the log. Not sure why. Here is what looks like the relevant part anyway, it seems that, though I have put the "tcp-established" in my /etc/criu/default and runc.conf files it isn't finding that configuration? (00.123855) inet: Dumping inet socket at 38 |
|
At the top of the log CRIU will say if it uses a configuration file. Not sure maybe your runc is too old. If you use RHEL 8.1 you should be able to test it with Podman: https://www.redhat.com/en/blog/container-migration-podman-rhel Podman has Seeing the complete log file would be important to know if CRIU sees your configuration file. |
|
Ah, looks like the config file was not used, it says "unable to get $HOME directory. I'll copy paste it in the bottom of this message, still unable to attach it. I know that it works on RHEL 8.1. I also dabbled with podman a bit but podman only supports one checkpoint, or at least it did back when I was looking into it about 6 months ago. For my use case, I need to be able to save multiple checkpoints so I have been using Docker. Log file paste: |
|
Using I still suspect that runc is somehow too old, because It is also strange that you get The $HOME message should not be a problem. |
|
Is that a new feature? I was taking that from this post containers/podman#2191 (comment) My runc.conf looks like: I tried it without the "skip-in-flight" option originally and it still was failing. |
It works differently, but you can create as many checkpoints as you want if you are using
Seeing "Will skip in-flight TCP connections" means that some configuration file is picked up. The difference is, that |
|
The only other configuration file in that directory is "default.conf" and I put the same two items in it because I wasn't sure which would get used. I'll try removing it. |
|
OK, I removed default.conf and now it doesn't show that it loaded any config files at the top of the log file. So, I am thinking that criu was loading default.conf, ignoring the Start of the log file: |
|
Can you try it with Podman, because that is known to work. |
|
That will take a bit to get to. My application and commands are all set up to work with Docker syntax and I've found podman syntax doesn't always line up exactly like it claims it will. It seems to me there should be another way to force runc to use the |
Yes. I mention multiple times that I assume that your runc used by docker is too old. Can you check which runc docker uses and which version it has. There are probably multiple runc binaries installed on your system. |
|
Well, sorry for the long delay. We didn't find any other runc's installed on the machine. We were however, able to find and install runc version 1.0.1-dev and that fixed the issue. Apparently 1.0.0-rc8-dev has some sort of issue. |
|
Perfect. Closing. |
I encounter this error when migrating a working project from RHEL8 to CentOS 8. Checkpoint save and restore works perfectly well until I enable tcp streams between my container and an external app, then, I get the error in the title:
Error response from daemon: Cannot checkpoint container <name>: runc did not terminate successfully: criu failed: type NOTIFY errno 0I've added "tcp-established" to both /etc/criu/runc.conf and /etc/criu/default.conf to no avail. I also tried adding "skip-in-flight" to those files as well. On my RHEL8 machine I was seeing a similar error. I believe the way we solved it was just getting the right version of Docker and CRIU installed. We can't get the packages to line up exactly the same on CentOS. At least, that is what our IT guy tells me. Even so, the versions are pretty close:
RHEL 8 versions:
CENTOS 8 versions:
I believe the relevant log section is:
{"level":"error","msg":"container is paused","time":"2020-06-19T10:53:27-06:00"} {"level":"error","msg":"criu failed: type NOTIFY errno 0\nlog file: /var/lib/containerd/io.containerd.runtime.v1.linux/moby/04c576d54cdade468215e05c48d7dca0db2392dc81fcbab07f3b7d871623e5f0/criu-work/dump.log\n","time":"2020-06-19T10:53:27-06:00"}The text was updated successfully, but these errors were encountered: