build: Ensure that binaries are run against their build-time ABI #897
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The subsequent commit will add an entry to create a /run/host symbolic link on the host that points to /, and it will require explicitly skipping some of the columns. Doing the same for the existing entry will make the file more readable. containers#821
|
It's necessary to copy |
|
Build failed.
|
HarryMichal
added
2. Under The Hood
HarryMichal
force-pushed
the
wip/rishi/run-host-glibc
branch
from
917c3ec to
b17f6c2
Compare
|
Build failed.
|
debarshiray
force-pushed
the
wip/rishi/run-host-glibc
branch
from
b17f6c2 to
9d0aa90
Compare
HarryMichal
force-pushed
the
wip/rishi/run-host-glibc
branch
from
9d0aa90 to
0f8987d
Compare
|
Build failed.
|
debarshiray
force-pushed
the
wip/rishi/run-host-glibc
branch
from
0f8987d to
cc357eb
Compare
|
Build failed.
|
debarshiray
force-pushed
the
wip/rishi/run-host-glibc
branch
from
cc357eb to
e15c359
Compare
|
Build failed.
|
debarshiray
force-pushed
the
wip/rishi/run-host-glibc
branch
from
e15c359 to
2f82dac
Compare
|
Build failed.
|
HarryMichal
added a commit
to HarryMichal/toolbox
that referenced
this pull request
Oct 21, 2021
HarryMichal
added a commit
to HarryMichal/toolbox
that referenced
this pull request
Oct 21, 2021
debarshiray
force-pushed
the
wip/rishi/run-host-glibc
branch
from
2f82dac to
022f818
Compare
The /usr/bin/toolbox binary is not only used to interact with toolbox
containers and images from the host. It's also used as the entry point
of the containers by bind mounting the binary from the host into the
container. This means that the /usr/bin/toolbox binary on the host must
also work inside the container, even if they have different operating
systems.
In the past, this worked perfectly well with the POSIX shell
implementation because it got intepreted by whichever /bin/sh was
available. However, the Go implementation, can run into ABI
compatibility issues because binaries built on newer toolchains aren't
meant to be run against older runtimes.
The previous approach [1] of restricting the versions of the glibc
symbols that are linked against isn't actually supported by glibc, and
breaks if the early process start-up code changes. This is seen in
glibc-2.34, which is used by Fedora 35 onwards, where a new version of
the __libc_start_main symbol [2] was added as part of some security
hardening:
$ objdump -T ./usr/bin/toolbox | grep GLIBC_2.34
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.34
__libc_start_main
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.34
pthread_detach
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.34
pthread_create
0000000000000000 DF *UND* 0000000000000000 GLIBC_2.34
pthread_attr_getstacksize
This means that /usr/bin/toolbox binaries built against glibc-2.34 on
newer Fedoras fail to run against older glibcs in older Fedoras.
Another option is to make the host's runtime available inside the
toolbox container and ensure that the binary always runs against it.
Luckily, almost all supported containers have the host's /usr available
at /run/host/usr. This is exploited by embedding RPATHs or RUNPATHs to
/run/host/usr/lib and /run/host/usr/lib64 in the binary, and changing
the path of the dynamic linker (ie., PT_INTERP) to the one inside
/run/host.
Unfortunately, there can only be one PT_INTERP entry inside the
binary, so there must be a /run/host on the host too. Therefore, a
/run/host symbolic link is created on the host that points to the
host's /.
Based on ideas from Alexander Larsson and Ray Strode.
[1] Commit 6ad9c63
containers#534
[2] glibc commit 035c012e32c11e84
https://sourceware.org/git/?p=glibc.git;a=commit;h=035c012e32c11e84
https://sourceware.org/bugzilla/show_bug.cgi?id=23323
containers#821
debarshiray
force-pushed
the
wip/rishi/run-host-glibc
branch
from
022f818 to
6063eb2
Compare
|
Build failed.
|
|
We still need to unbreak the CI but let's at least unblock the users now. |
HarryMichal
added a commit
to HarryMichal/toolbox
that referenced
this pull request
Oct 22, 2021
adjustmnets to the Toolbx binary that it requires presence of /run/host in both the host filesystem and the filesystem in a container. The presence of the directory is assured by systemd-tmpfiles by running it before the binary is started for the first time. This lead to the realization that the playbook structure is needlessly separated and can be unified as there is close to no benefit in keeping it separated. containers#898
HarryMichal
added a commit
to HarryMichal/toolbox
that referenced
this pull request
Oct 22, 2021
adjustmnets to the Toolbx binary that it requires presence of /run/host in both the host filesystem and the filesystem in a container. The presence of the directory is assured by systemd-tmpfiles by running it before the binary is started for the first time. This lead to the realization that the playbook structure is needlessly separated and can be unified as there is close to no benefit in keeping it separated. containers#898
HarryMichal
added a commit
to HarryMichal/toolbox
that referenced
this pull request
Oct 22, 2021
adjustmnets to the Toolbx binary that it requires presence of /run/host in both the host filesystem and the filesystem in a container. The presence of the directory is assured by systemd-tmpfiles by running it before the binary is started for the first time. This lead to the realization that the playbook structure is needlessly separated and can be unified as there is close to no benefit in keeping it separated. containers#898
HarryMichal
added a commit
to HarryMichal/toolbox
that referenced
this pull request
Oct 22, 2021
PR containers#897 made adjustmnets to the Toolbx binary that it requires presence of /run/host in both the host filesystem and the filesystem in a container. The presence of the directory is assured by systemd-tmpfiles by running it before the binary is started for the first time. For the run to be effective 'data/tmpfiles.d/toolbox.conf' has to be installed in a location visible to systemd-tmpfiles. Therefore, the call to 'systemd-tmpfiles --create' had to be placed after the install step.
HarryMichal
added a commit
to HarryMichal/toolbox
that referenced
this pull request
Oct 22, 2021
PR containers#897 made adjustmnets to the Toolbx binary that it requires presence of /run/host in both the host filesystem and the filesystem in a container. The presence of the directory is assured by systemd-tmpfiles by running it before the binary is started for the first time. For the run to be effective 'data/tmpfiles.d/toolbox.conf' has to be installed in a location visible to systemd-tmpfiles. Therefore, the call to 'systemd-tmpfiles --create' had to be placed after the install step.
HarryMichal
added a commit
to HarryMichal/toolbox
that referenced
this pull request
Oct 22, 2021
PR containers#897 made adjustmnets to the Toolbx binary that it requires presence of /run/host in both the host filesystem and the filesystem in a container. The presence of the directory is assured by systemd-tmpfiles by running it before the binary is started for the first time. For the run to be effective 'data/tmpfiles.d/toolbox.conf' has to be installed in a location visible to systemd-tmpfiles. Therefore, the call to 'systemd-tmpfiles --create' had to be placed after the install step. containers#898
HarryMichal
added a commit
that referenced
this pull request
Oct 22, 2021
PR #897 made adjustmnets to the Toolbx binary that it requires presence of /run/host in both the host filesystem and the filesystem in a container. The presence of the directory is assured by systemd-tmpfiles by running it before the binary is started for the first time. For the run to be effective 'data/tmpfiles.d/toolbox.conf' has to be installed in a location visible to systemd-tmpfiles. Therefore, the call to 'systemd-tmpfiles --create' had to be placed after the install step. #898
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The /usr/bin/toolbox binary is not only used to interact with toolbox
containers and images from the host. It's also used as the entry point
of the containers by bind mounting the binary from the host into the
container. This means that the /usr/bin/toolbox binary on the host must
also work inside the container, even if they have different operating
systems.
In the past, this worked perfectly well with the POSIX shell
implementation because it got intepreted by whichever /bin/sh was
available. However, the Go implementation, can run into ABI
compatibility issues because binaries built on newer toolchains aren't
meant to be run against older runtimes.
The previous approach [1] of restricting the versions of the glibc
symbols that are linked against isn't actually supported by glibc, and
breaks if the early process start-up code changes. This is seen in
glibc-2.34, which is used by Fedora 35 onwards, where a new version of
the __libc_start_main symbol [2] was added as part of some security
hardening:
This means that /usr/bin/toolbox binaries built against glibc-2.34 on
newer Fedoras fail to run against older glibcs in older Fedoras.
Another option is to make the host's runtime available inside the
toolbox container and ensure that the binary always runs against it.
Luckily, almost all supported containers have the host's /usr available
at /run/host/usr. This is exploited by embedding RPATHs or RUNPATHs to
/run/host/usr/lib and /run/host/usr/lib64 in the binary, and changing
the path of the dynamic linker (ie., PT_INTERP) to the one inside
/run/host.
Unfortunately, there can only be one PT_INTERP entry inside the
binary, so there must be a /run/host on the host too. Therefore, a
/run/host symbolic link is created on the host that points to the
host's /.
Based on ideas from Alexander Larsson and Ray Strode.
[1] Commit 6ad9c63
#534
[2] glibc commit 035c012e32c11e84
https://sourceware.org/git/?p=glibc.git;a=commit;h=035c012e32c11e84
https://sourceware.org/bugzilla/show_bug.cgi?id=23323
#821