Toolbox Error: failed to start container permission denied #734
Assignees
Labels
area/upstream
component/podman
component/toolbox
jira
for syncing to jira
kind/bug
needs/more-information
Comments
|
What does this show: The |
|
Also, what's the version of |
Actually, latest updates changed behaviour slightly. |
|
|
Looks like updated conmon in testing also does not help! |
|
Is toolbox broken beyond repair? Should one no longer use it? What exactly is the recommended way forward? Help! |
|
I'm currently investigating this one. In the meantime, you can use a "classic" Fedora container via podman. I'm also adding basic toolbox tests in coreos/fedora-coreos-config#862 so that we catch that earlier in the future. |
HarryMichal
added a commit
to HarryMichal/toolbox
that referenced
this issue
Feb 25, 2021
CoreOS recently made /boot read-only[0]. This caused an issue with starting containers because /boot was mounted only with option rslave but missed the ro option. This caused a permission issue. This scenario is very similar to the one with /usr on Fedora Silverblue. The solution for this is to check mount options of the path and check if it uses the rw option or ro and then add it to the mount options in the --volume option in 'podman create'. Fixes: coreos/fedora-coreos-tracker#734 coreos/fedora-coreos-config@1de21ff
|
I believe this is caused by coreos/fedora-coreos-config@1de21ff . This change in CoreOS makes I just proposed a fix for this upstream: containers/toolbox#712. I tried it on an instance of CoreOS and it seems to fix the issue. @ziswiler, @travier, would you be so kind and tried the fix, too? I'm afraid the fix will not fix your existing toolboxes because this is an error in container configuration and that can not be changed once a container has been created. |
|
Thanks for the investigation. I'm wondering if that's the full issue as on a stable FCOS ( thus I'm wondering why we are not having this issue on stable. |
|
Just tried the same command and I get a bit different output: Maybe the two security-related options are causing the problem? My CoreOS instance version is also |
|
Great, thank you very much!
Please excuse my ignorance but how exactly would I go about testing any such? As for the existing toolbox' mount options that looks indeed the same for me: |
HarryMichal
added a commit
to HarryMichal/toolbox
that referenced
this issue
Mar 1, 2021
CoreOS recently made /boot read-only[0]. This caused an issue with starting containers because /boot was mounted only with option rslave but missed the ro option. This caused a permission issue. This scenario is very similar to the one with /usr on Fedora Silverblue. The solution for this is to check mount options of the path and check if it uses the rw option or ro and then add it to the mount options in the --volume option in 'podman create'. Fixes: coreos/fedora-coreos-tracker#734 [0] coreos/fedora-coreos-config@1de21ff containers#712
HarryMichal
added a commit
to containers/toolbox
that referenced
this issue
Mar 1, 2021
CoreOS recently made /boot read-only[0]. This caused an issue with starting containers because /boot was mounted only with option rslave but missed the ro option. This caused a permission issue. This scenario is very similar to the one with /usr on Fedora Silverblue. The solution for this is to check mount options of the path and check if it uses the rw option or ro and then add it to the mount options in the --volume option in 'podman create'. Fixes: coreos/fedora-coreos-tracker#734 [0] coreos/fedora-coreos-config@1de21ff #712
|
I can not reproduce that anymore either on stable or testing. Feel free to re-open if that's still the case for you. |
|
Well, for me on latest Fedora CoreOS Testing it still does NOT work: |
|
I've just given this a try on a fresh testing image and this worked for me. Can you give us the output of |
|
Closing this one until we get more info. |
I am not exactly sure what further information you are looking for. It all worked just fine until around X-mas. Since then I have not seen any toolbox working any longer whatever I tried... |
|
As your UID and GID do not match and that may have an impact with rootless podman, can you try to run the container with |
You mean in addition to the
Unfortunately, this being a bare-metal instance this is not how that user got created. I believe regular useradd was used for that. But again, it worked just fine until around X-mas. Something in that whole tooling must have changed... |
|
Can you try |
I guess that does not work as it does not get that far: However, attach suggests that it really has to do with that user/group ID stuff. But, again, that would just be a regression as that all used to work just fine, not? |
|
OK, I can confirm that this has indeed to do with my regular user's UID and GID not being the same. It does work with a different user account that has those both set to the same. As this used to work just fine before around X-mas time I would consider this a regression. What do you think? |
|
Can you open a bug report upstream in podman and link it here? Thanks! |
travier
added
component/podman
and removed
needs/investigation
needs/more-information
labels
In March we merged this commit: I wonder if that's causing problems for you. To be honest, the commit is a bit dubious. See the second paragraph of the commit message. However, the NixOS folks wanted it because it helped them get Toolbox running on NixOS hosts, so I didn't want to block them indefinitely. |
|
Hm, that thing still does not work even after I meanwhile changed to such wired uid:gid being the same number kinda setup: Ah, yeah, before I forget it, that's on latest testing: What could be the issue now? Anyway, I am tempted to throw it all out the window and dig out my good oldé C64 again. That thing used to rock! 🤣 |
|
Have you try creating a new toolbox while running under your new uid/gid matching user? Could you also check the |
Yes, of course. I even removed any and all containers so
Sure, I guess by the later you mean |
|
Actually, one strange thing I noticed is that them files of the containers storage overlay are owned by Any idea what I may have messed up? |
|
On a fresh image: The ID overlap may be the root of this issue:
|
|
Closing this one pending more information. |
|
@travier You can find more in #containers/podman#12986 |
|
@travier I found something very similar in the RHEL bugzilla but proposed solution is not applicable to Fedora 35 - the versions container-selinux don't match. From the recent log I see that CRun when it is invoked by Conmon for the rootless user tries to run container's entry-point from /var/run... I created the new image using buildah with my own entry-point script in /var/bin and it doesn't start too. From the SELinux point of view user's tmpfs and root's tmpfs are labeled differently and also made available differently. Indeed, root's /var/run and user's /run/user/uid are different things. I would like to look at Fedora's SELinux module from the corresponding container-selinux. I guess that either UID mapping is needed or vector container_runtime or run-labelled exec entry-point script execution by CRun should be allowed. |
|
Please don't ping individuals and open another issue with updated details about your current issue. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
Entering a toolbox on Fedora fails yet again.
Reproduction steps
Steps to reproduce the behavior:
Expected behavior
One should be inside the toolbox container.
Actual behavior
Error occurs.
System details
Ignition config
Really nothing ignition specific as it worked before, nothing changed in such ignition respect.
Additional information
Add any other information about the problem here.
And full logfile:
The text was updated successfully, but these errors were encountered: