No status provided on response: unknown #813
Comments
|
I’ve managed to fix this by reinstalling another version of runc and then reinstalling docker-ce My docker version changed to: I think only the containerd version is different. |
|
Same error here after this morning automatic upgrade of Ubuntu Any
|
|
@vietthang207 thanks for the quickfix. To clarify a bit, I had to:
|
I've fixed this, thanks. |
|
It worked for me too, thanks. This means that docker is broken on every autoupdating Ubuntu 16.04 server and desktop today, unless this workaround is applied? |
|
The fix doesn't work for me on Ubuntu 18.04. I see from apt-get logs, the re-installation of docker-ce removes again runs and installs again containerd.io. Does anyone know if there's a precise version (as the one proposed by @vietthang207 for 16.04) of runc to be installed on 18.04? |
|
Run this command to list the available versions of The output I got on a 18.04 server is One of them should be the right one for you. Apparently that server didn't have any problem today. |
|
What worked for me on 16.04 was installing older # apt-cache madison docker-ce | grep 18.09.2
docker-ce | 5:18.09.2~3-0~ubuntu-xenial | https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
# apt-cache madison containerd.io | grep 1.2.6
containerd.io | 1.2.6-3 | https://download.docker.com/linux/ubuntu xenial/stable amd64 Packages
# apt-get install docker-ce=5:18.09.2~3-0~ubuntu-xenial docker-ce-cli=5:18.09.2~3-0~ubuntu-xenial containerd.io=1.2.6-3
<...>
# $ docker run hello-world
<...>
Hello from Docker!
<...>(You might wanna go with newer version combinations if they work for you.) Note to self: in the future, pin not only # apt-mark hold docker-ce
# apt-mark hold docker-ce-cli
# apt-mark hold containerd.ioDisabling unattended upgrades is not the worst idea either. |
|
Installing the prior version of containerd using:
worked for me. No need to reinstall docker-ce that way. |
|
@ventsip I think it is just solved part of some problem, with your re-installation, my docker-cli runs ok now, but it broken the nvidia-docker cmd, now if I run nvidia-docker, it returns Any suggestions? UPDATE: I think it is the upgrade with nvidia-docker-cli as well, if I changed the cmdline from to It works as expected. |
|
I had this error today while building a container. It was just after upgrading my system on Manjaro (Arch). I probably got a docker update. Restarting the docker service fixed the issue: sudo systemctl restart docker |
it works also for me. No docker reinstall needed. |
|
For me just restarting the docker service solved the problem. |
lucky guy :) |
|
#813 (comment) This helped me indeed :) |
|
I also got this issue, after latest update on Ubuntu 16.04.6 LTS |
Thanks for this! Fixed it for me. :) |
also work for me. After update the containerd, need to restart the docker daemon. extra info: |
|
Hi folks,
Are you sure you intended to tag me to this message?
Thanks,
Ventsi
… On Oct 11, 2019, at 7:30 PM, Teng Fu ***@***.***> wrote:
@ventsip <https://github.com/ventsip> I think it is just solved part of some problem, with your re-installation, my docker-cli runs ok now, but it broken the nvidia-docker cmd, now if I run nvidia-docker, it returns
sudo: nvidia-docker: command not found
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#813?email_source=notifications&email_token=AFBTRHTF5HW337CYWDF5GVTQOCS2TA5CNFSM4I7JS37KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBAQY4Y#issuecomment-541133939>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFBTRHSJ7K3PP37QF2F6FETQOCS2TANCNFSM4I7JS37A>.
|
|
As per petiepooo comment downgrading to prior version of containerd.io and restarting docker service worked for me (Ubuntu 16.04.6 LTS) |
@ventsip Sorry, I think I got it wrong, I wish to @ vietthang207 , GitHub suggested you but I forgot to check. |
|
... i just followed the instructions here https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-16-04 which updated my docker to 19.03.03 from 19.03.01 or some such, and now it's working at least for hello-world, and my docker containers are starting to build, without failing mysteriously after the FROM command. |
|
on archlinux, with docker 19.03.2 downgrading containerd from 1.3.0 to 1.2.8 worked again |
|
the following just worked for me after upgrading to docker 19.03.3. ub18.04 |
How you did it? I faced the same issue in Manjaro. |
Thanks. It worked. |
|
I also have the same issue on my Ubuntu 16.04. After I reinstall the |
FWIW, I discovered it when I went to build a container -- after it downloaded the base image for the container, it would hit the first RUN command, and then die with the same error. I don't normally use docker on the machine for much of anything, but I had used it last week to run a test of something, and then went I went back to it the other day, was when this started showing up. |
This did the trick for me on Ubuntu 16.04, but I also had to run |
|
Hello All - I think the issue is confirmed at this point. Can I request that we stop providing a solution for each and every thing that "worked for me". Don't get me wrong, these were helpful, but at this point, I don't think this is helping the devs triage the issue. Thanks! |
|
One thing that would be helpful for all of us who have encountered it, and
those in the future, is to know whether a fix is likely to come as 1.2.10
but with a higher packaging numeric value, or whether it will be 1.2.X
where X > 10.
…On Fri, Oct 18, 2019 at 4:02 PM adawalli ***@***.***> wrote:
Hello All - I think the issue is confirmed at this point. Can I request
that we stop providing a solution for each and every thing that "worked for
me". Don't get me wrong, these were helpful, but at this point, I don't
think this is helping the devs triage the issue. Thanks!
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#813?email_source=notifications&email_token=AAJQQRZPIKUYLN7NVVJKOIDQPIP7BA5CNFSM4I7JS37KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBV67VY#issuecomment-543944663>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAJQQR2RBPU545NKRRPFKVLQPIP7BANCNFSM4I7JS37A>
.
--
Chris Cleeland
|
|
I also got bit by this, on a Friday afternoon, while at an event...note to self: don't do this. |
|
I understand this is a bug in Ubuntu packaging, but can someone explain why, 8 days after this was reported, upgrading an Ubuntu machine still will break? |
|
@alexanderkjeldaas the Package is for Ubuntu no from Ubuntu, docker-ce is an additional repository you added to your system. The responsibility for that repo is at Docker Inc. And as it is only the community edition it looks like it is not treated with highest priority. |
The team has been working on this issue for the last 3 days; even though the problem was located, finding the right solution turned out to have some issues (fixing the problem caused other up/downgrade scenarios to be broken), so all of those had to be re-verified. A fix was found, and the @alexanderkjeldaas were you still running into this issue when upgrading from containerd.io 1.2.5/1.2.6 to 1.2.10-3 ? |
|
Note that a workaround for those that got stuck after upgrading to the 1.2.10-2 package on Ubuntu 16.04 is to do a |
|
Today, docker service was shut down after unattended-upgrades updated It looks like now the containerd.io package restarts its service, but introduces another (even more serious) issue that leaves the docker service stopped.
|
|
|
|
@thaJeztah It looks like it is dependent on the update order. on hosts where docker-ce is updated after containerd.io, the service and containers were running. on hosts where containerd.io was updated after docker-ce, the service is down and on one host I had now to manually kill and remove the containers before i could start them again. restarting the services was not enough. i had to manually kill the services (
|
|
Here the full output of what I had to to on another server that was down: https://pastebin.com/Xyu7TypW |
|
No direct clue what's causing the issue, looking through those logs (logs with some empty lines cleaned up here: docker_update_issue.log) The shims for those containers are running; But when trying to use the container, it attempts to create a new socket, which fails, likely fails because there's already a shim running; After restarting docker, something attempts to recreate the container in runc, and fails because it already exists |
My system is working now with the downgrade and restart of containerd (I think). That's awesome. I don't think I'll try the upgrade right now as it's working fine now. My point was, that even if downgrading was broken, wouldn't it make sense to pull the latest package to avoid new systems getting into trouble? As you said, there was no solution to the systems that were broken already during this time, but by pulling the packages it would reduce the number of new systems getting into trouble. So that's why I was surprised that I upgraded and found myself with a broken system 3 (?) days after the issue was reported/detected. Maybe that could be added as a point to the resolution protocol for issues like this. I know that in an organization like yours, it's probably different people or teams that do fixes vs packaging, so some communication between them is needed to do things like that. |
|
We also had this issue a few days ago on an Ubuntu 16.04.6 LTS server which was running Docker 18.09.6. We ran updates on Oct/08, and that brought docker-ce-cli up to 19.03.2. But oddly enough, this only updated docker-ce-cli, not docker-ce. We did not realize this at the time, as we restarted existing containers and everything worked fine. Days later there was the sudo vulnerability, so we updated again (Oct/15,) that brought docker-ce-cli from 19.03.2 to 19.03.3, but again, only docker-ce-cli, not docker-ce which remained untouched. We did not realize this, and we did not see any trouble after this update either. Only when developers on that server tried to pull and run new images, we got the OP error. We found this page after that, and we decided to simply bring our docker-ce-cli to the same version our docker-ce still had, which was 18.09.6. We did so, and the problem persisted even if restarting the docker daemon. Only after completely rebooting the server, everything was back to normal. |
|
To address the original issue reported, we've released containerd.io 1.2.10-3. We're taking our lessons learned from this past release to add more checks to validate upgrade scenario before we release newer versions of containerd.io. |
|
Had a look at #813 (comment) and it looks like this is a separate issue to be tracked in http://github.com/containerd/containerd/issues |
|
DISTRIB_CODENAME=xenial apt search docker-ce docker-ce-cli/xenial,now 5:19.03.4 containerd.io 1.2.10-3 or containerd.io 1.2.10-2 not important. Process: 6974 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE) 07 14:06:46 OrangePi systemd[1]: Failed to start Docker Application Container Engine. I can't to install docker! |
|
In case someone uses Centos 7 or RHEL 7, here's a simple way to work around this bug until version 1.2.10+ of |
|
We seem to be hitting this issue right now in Debian unstable: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=956502 @thaJeztah At the time you said:
I know it's a bit late to ask, but is there a chance to know what was the issue, and what was the fix? Thanks! (it would have been tremendously useful if the source package for |
|
@elboulangero the fix was in docker/containerd-packaging#113, which was related to a change in newer versions of debhelper causing the restart after upgrade to not being done. |
|
Thanks! Somehow I couldn't find the containerd-packaging repo when I was looking for it. Everything is fixed now. Thanks again and sorry for the noise! |
|
It was a private repository until recently, so that likely explains 😅 |
|
For Cent OS users. Additionally to #813 (comment), I faced this case, and resolved with the following commands. |
and others
Expected behavior
Actual behavior
Steps to reproduce the behavior
Output of
docker version:Output of
docker info:Additional environment details (AWS, VirtualBox, physical, etc.)
AWS
The text was updated successfully, but these errors were encountered: