-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
runc run failure logging when the binary is on noexec fs #3520
Comments
This comment was marked as outdated.
This comment was marked as outdated.
OK this only happens for Ended up with this CR: https://go-review.googlesource.com/c/go/+/414824 |
oh! nice find |
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Jun 30, 2022
When starting a new container, and the very last step of executing of a user process fails (last lines of (*linuxStandardInit).Init), it is too late to print a proper error since both the log pipe and the init pipe are closed. This is partially mitigated by using exec.LookPath() which is supposed to say whether we will be able to execute or not. Alas, it fails to do so when the binary to be executed resides on a filesystem mounted with noexec flag. A workaround is to use access(2) with X_OK flag. Fixes opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
changed the title
logging vs podman volume
runc run failure logging when the binary is on noexec fs
Jun 30, 2022
kolyshkin
added a commit
to kolyshkin/podman
that referenced
this issue
Jun 30, 2022
A newer runc (v1.1.x) changed the error message again, so simplify the check to look for a common denominator of all these errors. Note that due to an entirely different issue [1] the test still fails when using runc, but this is being fixed elsewhere ([2], [3]). This will presumably be fixed in runc 1.1.4 and runc 1.2.0. [1] opencontainers/runc#3520 [2] opencontainers/runc#3522 [3] https://go-review.googlesource.com/c/go/+/414824 Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Jul 28, 2022
When starting a new container, and the very last step of executing of a user process fails (last lines of (*linuxStandardInit).Init), it is too late to print a proper error since both the log pipe and the init pipe are closed. This is partially mitigated by using exec.LookPath() which is supposed to say whether we will be able to execute or not. Alas, it fails to do so when the binary to be executed resides on a filesystem mounted with noexec flag. A workaround would be to use access(2) with X_OK flag. Alas, it is not working when runc itself is a setuid (or setgid) binary. In this case, faccessat2(2) with AT_EACCESS can be used, but it is only available since Linux v5.8. So, use faccessat2(2) with AT_EACCESS if available. If not, fall back to access(2) for non-setuid runc, and do nothing for setuid runc (as there is nothing we can do). Note that this check if in addition to whatever exec.LookPath does. Fixes opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit 957d97b) Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Jul 28, 2022
When starting a new container, and the very last step of executing of a user process fails (last lines of (*linuxStandardInit).Init), it is too late to print a proper error since both the log pipe and the init pipe are closed. This is partially mitigated by using exec.LookPath() which is supposed to say whether we will be able to execute or not. Alas, it fails to do so when the binary to be executed resides on a filesystem mounted with noexec flag. A workaround would be to use access(2) with X_OK flag. Alas, it is not working when runc itself is a setuid (or setgid) binary. In this case, faccessat2(2) with AT_EACCESS can be used, but it is only available since Linux v5.8. So, use faccessat2(2) with AT_EACCESS if available. If not, fall back to access(2) for non-setuid runc, and do nothing for setuid runc (as there is nothing we can do). Note that this check if in addition to whatever exec.LookPath does. Fixes opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit 957d97b) Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Feb 28, 2023
Since this commit was made, a few things happened: - a similar functionality appeared in go 1.20 [1] - a bug in runc was found [2], which also affects go [3] - the bug was fixed in go 1.21 [4] and 1.20.2 [5] - a similar fix was made to x/sys/unix.Faccessat [6] Revert commit 957d97b so we can fix the bug [2] when go > 1.21.1 is used. Note that this will reintroduce the older bug [7] when the older go version is used, but since this is a minor bug which will be fixed once everyone switches to a recent go version, let's keep things simple and not introduce any complex code here. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Feb 28, 2023
Since this commit was made, a few things happened: - a similar functionality appeared in go 1.20 [1], so the issue mentioned in the comment (being removed) is no longer true; - a bug in runc was found [2], which also affects go [3]; - the bug was fixed in go 1.21 [4] and 1.20.2 [5]; - a similar fix was made to x/sys/unix.Faccessat [6]. Revert commit 957d97b so we can fix the bug [2] when go > 1.21.1 is used. Note that this will reintroduce the older bug [7] when the older go version is used, but since this is a minor bug which will be fixed once everyone switches to a recent go version, let's keep things simple and not introduce any complex code here. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Mar 9, 2023
Since commit 957d97b was made to fix issue [7], a few things happened: - a similar functionality appeared in go 1.20 [1], so the issue mentioned in the comment (being removed) is no longer true; - a bug in runc was found [2], which also affects go [3]; - the bug was fixed in go 1.21 [4] and 1.20.2 [5]; - a similar fix was made to x/sys/unix.Faccessat [6]. The essense of [2] is, even if a (non-root) user that the container is run as does not have execute permission bit set for the executable, it should still work in case runc has the CAP_DAC_OVERRIDE capability set. To fix this [2] without reintroducing the older bug [7]: - drop own Eaccess implementation; - use the one from x/sys/unix for Go 1.19 (depends on [6]); - do not use anything when Go 1.20+ is used. NOTE it is virtually impossible to fix the bug [2] when Go 1.20 or Go 1.20.1 is used because of [3]. A test case is added by a separate commit. Fixes: opencontainers#3715. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Mar 9, 2023
Since commit 957d97b was made to fix issue [7], a few things happened: - a similar functionality appeared in go 1.20 [1], so the issue mentioned in the comment (being removed) is no longer true; - a bug in runc was found [2], which also affects go [3]; - the bug was fixed in go 1.21 [4] and 1.20.2 [5]; - a similar fix was made to x/sys/unix.Faccessat [6]. The essense of [2] is, even if a (non-root) user that the container is run as does not have execute permission bit set for the executable, it should still work in case runc has the CAP_DAC_OVERRIDE capability set. To fix this [2] without reintroducing the older bug [7]: - drop own Eaccess implementation; - use the one from x/sys/unix for Go 1.19 (depends on [6]); - do not use anything when Go 1.20+ is used. NOTE it is virtually impossible to fix the bug [2] when Go 1.20 or Go 1.20.1 is used because of [3]. A test case is added by a separate commit. Fixes: opencontainers#3715. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Mar 16, 2023
Since commit 957d97b was made to fix issue [7], a few things happened: - a similar functionality appeared in go 1.20 [1], so the issue mentioned in the comment (being removed) is no longer true; - a bug in runc was found [2], which also affects go [3]; - the bug was fixed in go 1.21 [4] and 1.20.2 [5]; - a similar fix was made to x/sys/unix.Faccessat [6]. The essense of [2] is, even if a (non-root) user that the container is run as does not have execute permission bit set for the executable, it should still work in case runc has the CAP_DAC_OVERRIDE capability set. To fix this [2] without reintroducing the older bug [7]: - drop own Eaccess implementation; - use the one from x/sys/unix for Go 1.19 (depends on [6]); - do not use anything when Go 1.20+ is used. NOTE it is virtually impossible to fix the bug [2] when Go 1.20 or Go 1.20.1 is used because of [3]. A test case is added by a separate commit. Fixes: opencontainers#3715. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Mar 27, 2023
Since commit 957d97b was made to fix issue [7], a few things happened: - a similar functionality appeared in go 1.20 [1], so the issue mentioned in the comment (being removed) is no longer true; - a bug in runc was found [2], which also affects go [3]; - the bug was fixed in go 1.21 [4] and 1.20.2 [5]; - a similar fix was made to x/sys/unix.Faccessat [6]. The essense of [2] is, even if a (non-root) user that the container is run as does not have execute permission bit set for the executable, it should still work in case runc has the CAP_DAC_OVERRIDE capability set. To fix this [2] without reintroducing the older bug [7]: - drop own Eaccess implementation; - use the one from x/sys/unix for Go 1.19 (depends on [6]); - do not use anything when Go 1.20+ is used. NOTE it is virtually impossible to fix the bug [2] when Go 1.20 or Go 1.20.1 is used because of [3]. A test case is added by a separate commit. Fixes: opencontainers#3715. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Apr 6, 2023
Since commit 957d97b was made to fix issue [7], a few things happened: - a similar functionality appeared in go 1.20 [1], so the issue mentioned in the comment (being removed) is no longer true; - a bug in runc was found [2], which also affects go [3]; - the bug was fixed in go 1.21 [4] and 1.20.2 [5]; - a similar fix was made to x/sys/unix.Faccessat [6]. The essense of [2] is, even if a (non-root) user that the container is run as does not have execute permission bit set for the executable, it should still work in case runc has the CAP_DAC_OVERRIDE capability set. To fix this [2] without reintroducing the older bug [7]: - drop own Eaccess implementation; - use the one from x/sys/unix for Go 1.19 (depends on [6]); - do not use anything when Go 1.20+ is used. NOTE it is virtually impossible to fix the bug [2] when Go 1.20 or Go 1.20.1 is used because of [3]. A test case is added by a separate commit. Fixes: opencontainers#3715. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit 8491d33) Signed-off-by: Kir Kolyshkin <[email protected]>
kolyshkin
added a commit
to kolyshkin/runc
that referenced
this issue
Apr 7, 2023
Since commit 957d97b was made to fix issue [7], a few things happened: - a similar functionality appeared in go 1.20 [1], so the issue mentioned in the comment (being removed) is no longer true; - a bug in runc was found [2], which also affects go [3]; - the bug was fixed in go 1.21 [4] and 1.20.2 [5]; - a similar fix was made to x/sys/unix.Faccessat [6]. The essense of [2] is, even if a (non-root) user that the container is run as does not have execute permission bit set for the executable, it should still work in case runc has the CAP_DAC_OVERRIDE capability set. To fix this [2] without reintroducing the older bug [7]: - drop own Eaccess implementation; - use the one from x/sys/unix for Go 1.19 (depends on [6]); - do not use anything when Go 1.20+ is used. NOTE it is virtually impossible to fix the bug [2] when Go 1.20 or Go 1.20.1 is used because of [3]. A test case is added by a separate commit. Fixes: opencontainers#3715. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers#3520 Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit 8491d33) Signed-off-by: Kir Kolyshkin <[email protected]>
dims
pushed a commit
to dims/libcontainer
that referenced
this issue
Oct 19, 2024
When starting a new container, and the very last step of executing of a user process fails (last lines of (*linuxStandardInit).Init), it is too late to print a proper error since both the log pipe and the init pipe are closed. This is partially mitigated by using exec.LookPath() which is supposed to say whether we will be able to execute or not. Alas, it fails to do so when the binary to be executed resides on a filesystem mounted with noexec flag. A workaround would be to use access(2) with X_OK flag. Alas, it is not working when runc itself is a setuid (or setgid) binary. In this case, faccessat2(2) with AT_EACCESS can be used, but it is only available since Linux v5.8. So, use faccessat2(2) with AT_EACCESS if available. If not, fall back to access(2) for non-setuid runc, and do nothing for setuid runc (as there is nothing we can do). Note that this check if in addition to whatever exec.LookPath does. Fixes opencontainers/runc#3520 Signed-off-by: Kir Kolyshkin <[email protected]>
dims
pushed a commit
to dims/libcontainer
that referenced
this issue
Oct 19, 2024
Since commit e2100c8 was made to fix issue [7], a few things happened: - a similar functionality appeared in go 1.20 [1], so the issue mentioned in the comment (being removed) is no longer true; - a bug in runc was found [2], which also affects go [3]; - the bug was fixed in go 1.21 [4] and 1.20.2 [5]; - a similar fix was made to x/sys/unix.Faccessat [6]. The essense of [2] is, even if a (non-root) user that the container is run as does not have execute permission bit set for the executable, it should still work in case runc has the CAP_DAC_OVERRIDE capability set. To fix this [2] without reintroducing the older bug [7]: - drop own Eaccess implementation; - use the one from x/sys/unix for Go 1.19 (depends on [6]); - do not use anything when Go 1.20+ is used. NOTE it is virtually impossible to fix the bug [2] when Go 1.20 or Go 1.20.1 is used because of [3]. A test case is added by a separate commit. Fixes: #3715. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers/runc#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers/runc#3520 Signed-off-by: Kir Kolyshkin <[email protected]>
dims
pushed a commit
to dims/libcontainer
that referenced
this issue
Oct 19, 2024
When starting a new container, and the very last step of executing of a user process fails (last lines of (*linuxStandardInit).Init), it is too late to print a proper error since both the log pipe and the init pipe are closed. This is partially mitigated by using exec.LookPath() which is supposed to say whether we will be able to execute or not. Alas, it fails to do so when the binary to be executed resides on a filesystem mounted with noexec flag. A workaround would be to use access(2) with X_OK flag. Alas, it is not working when runc itself is a setuid (or setgid) binary. In this case, faccessat2(2) with AT_EACCESS can be used, but it is only available since Linux v5.8. So, use faccessat2(2) with AT_EACCESS if available. If not, fall back to access(2) for non-setuid runc, and do nothing for setuid runc (as there is nothing we can do). Note that this check if in addition to whatever exec.LookPath does. Fixes opencontainers/runc#3520 Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit 957d97b) Signed-off-by: Kir Kolyshkin <[email protected]>
dims
pushed a commit
to dims/libcontainer
that referenced
this issue
Oct 19, 2024
Since commit 957d97b was made to fix issue [7], a few things happened: - a similar functionality appeared in go 1.20 [1], so the issue mentioned in the comment (being removed) is no longer true; - a bug in runc was found [2], which also affects go [3]; - the bug was fixed in go 1.21 [4] and 1.20.2 [5]; - a similar fix was made to x/sys/unix.Faccessat [6]. The essense of [2] is, even if a (non-root) user that the container is run as does not have execute permission bit set for the executable, it should still work in case runc has the CAP_DAC_OVERRIDE capability set. To fix this [2] without reintroducing the older bug [7]: - drop own Eaccess implementation; - use the one from x/sys/unix for Go 1.19 (depends on [6]); - do not use anything when Go 1.20+ is used. NOTE it is virtually impossible to fix the bug [2] when Go 1.20 or Go 1.20.1 is used because of [3]. A test case is added by a separate commit. Fixes: #3715. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers/runc#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers/runc#3520 Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit 8491d33) Signed-off-by: Kir Kolyshkin <[email protected]>
dims
pushed a commit
to dims/libcontainer
that referenced
this issue
Oct 19, 2024
When starting a new container, and the very last step of executing of a user process fails (last lines of (*linuxStandardInit).Init), it is too late to print a proper error since both the log pipe and the init pipe are closed. This is partially mitigated by using exec.LookPath() which is supposed to say whether we will be able to execute or not. Alas, it fails to do so when the binary to be executed resides on a filesystem mounted with noexec flag. A workaround would be to use access(2) with X_OK flag. Alas, it is not working when runc itself is a setuid (or setgid) binary. In this case, faccessat2(2) with AT_EACCESS can be used, but it is only available since Linux v5.8. So, use faccessat2(2) with AT_EACCESS if available. If not, fall back to access(2) for non-setuid runc, and do nothing for setuid runc (as there is nothing we can do). Note that this check if in addition to whatever exec.LookPath does. Fixes opencontainers/runc#3520 Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit a32275ab054d34d4668383dd57bc7762863b4174) Signed-off-by: Kir Kolyshkin <[email protected]>
dims
pushed a commit
to dims/libcontainer
that referenced
this issue
Oct 19, 2024
Since commit a32275ab054d34d46683 was made to fix issue [7], a few things happened: - a similar functionality appeared in go 1.20 [1], so the issue mentioned in the comment (being removed) is no longer true; - a bug in runc was found [2], which also affects go [3]; - the bug was fixed in go 1.21 [4] and 1.20.2 [5]; - a similar fix was made to x/sys/unix.Faccessat [6]. The essense of [2] is, even if a (non-root) user that the container is run as does not have execute permission bit set for the executable, it should still work in case runc has the CAP_DAC_OVERRIDE capability set. To fix this [2] without reintroducing the older bug [7]: - drop own Eaccess implementation; - use the one from x/sys/unix for Go 1.19 (depends on [6]); - do not use anything when Go 1.20+ is used. NOTE it is virtually impossible to fix the bug [2] when Go 1.20 or Go 1.20.1 is used because of [3]. A test case is added by a separate commit. Fixes: #3715. [1] https://go-review.googlesource.com/c/go/+/414824 [2] opencontainers/runc#3715 [3] https://go.dev/issue/58552 [4] https://go-review.googlesource.com/c/go/+/468735 [5] https://go-review.googlesource.com/c/go/+/469956 [6] https://go-review.googlesource.com/c/sys/+/468877 [7] opencontainers/runc#3520 Signed-off-by: Kir Kolyshkin <[email protected]> (cherry picked from commit b7fff20cb91cc48e3995a86fedfa10d7baa17c4c) Signed-off-by: Kir Kolyshkin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When
runc run
is called by podman, podman gathers run output, and if it contains "permission denied", exits with code of 126.As demonstrated:
Now, for some reason yet unknown to me, if the non-executable binary is on volume, runc fails to propagate the error back to podman.
Here's a repro with a slightly modified runc:
here is what happens normally:
Now, if we add
noexec
flag to the bind mount:This is some kind of a race upon exit. Can't figure out any more details at the moment¸will continue tomorrow.
The text was updated successfully, but these errors were encountered: