Generate synthetic object file to ensure all exported and used symbols participate in the linking #95604
Conversation
rustbot
added
the
T-compiler
rust-highfive
added
the
S-waiting-on-review
|
One reason why I like this approach more than #95363 is that I have no idea how widely linker script files like A "root" object file, on another hand, is as portable as it gets, with the same logic being reusable for very different targets. |
This comment has been minimized.
This comment has been minimized.
|
Yeah this seems like a cleaner approach. This should fix the This doesn't fix #47384 fully however as it still requires |
What do you mean? This certainly fixes the issue described by the OP in #47384. |
|
Ah, the rust-audit case does require a custom linker script or |
This comment has been minimized.
This comment has been minimized.
|
#47384 discusses so many different things that I'd like to set some agreements on what specifically this PR is supposed to assume and fix.
|
|
@nbdd0121
I'll be able to approve the first PR almost immediately, and it will then unblock #95818. |
No. This PR does not change the meaning of
No. Neither of these are exported. The export symbol list is not changed at all.
See above. |
As I understand the |
|
The |
|
There is an "exported_symbols" query and a "exported_symbols" function in linker.rs that produces the final exported symbols. They are actually different things; the first contains all reachables, latter does a filter to the former to get an actual list for export. |
Yes, but they shouldn't have any exported level at all (unless they are exported for other reasons unrelated to
The list of symbols to filter in |
"Rust" level export will be exported in I don't see why #[used]
static A: u32 = 1;shouldn't be exported while static A: u32 = 1;
#[inline]
pub fn foo() -> &u32 { &A }should. |
|
In a sense I don't treat "Rust" export level as truly being exports; it's basically just an export level for "everything that might get used" and I view it as an implementation detail to make dylibs work. |
|
Ok, point understood. I agree that in practice it's not too important to prune such used symbols from dylib exports, and we can skip it if it helps to simplify the implementation or fix other more important issues sooner. I'd still consider it as a rough edge of the implementation, because when spec / mental model say one thing and the implementation do another it often leads to misunderstanding and mess. |
|
The issue 1 ( |
|
Otherwise LGTM, no need to split into two PRs (#95604 (comment)) anymore. |
petrochenkov
added
S-waiting-on-author
Fix e_flags for 32-bit MIPS targets in generated object file In rust-lang#95604 the compiler started generating a temporary symbols.o which is added to the linker invocation. This object file has an `e_flags` which is invalid for 32-bit MIPS targets. Even though symbols.o doesn't contain code, linking these targets with [lld fails](https://github.com/llvm/llvm-project/blob/main/lld/ELF/Arch/MipsArchTree.cpp#L76-L79) with ``` rust-lld: error: foo-cgu.0.rcgu.o: ABI 'o32' is incompatible with target ABI 'n64' ``` because it omits the ABI bits (`EF_MIPS_ABI_O32`) so lld assumes it's using the N64 ABI. This breaks linking on nightly for the out-of-tree [mipsel-sony-psx target](ayrtonm/psx-sdk-rs#9), the builtin mipsel-sony-psp target (cc `@overdrivenpotato)` and probably any other 32-bit MIPS target using lld. This PR sets the ABI in `e_flags` to O32 since that's the only ABI for 32-bit MIPS that LLVM supports. It also sets other `e_flags` bits based on the target to avoid similar issues with the object file arch and PIC. I had to bump the object crate version since some of these constants were [added recently](gimli-rs/object#433). I'm not sure if this PR needs a test, but I can confirm that it fixes the linking issue on both targets I mentioned.
Fixes linker errors in an application that is linked to `r3_port_riscv` and does not use the `riscv-rt`-compatible startup routine provided by `r3_port_riscv::use_rt!`. Rust implements `rlib`s as static libraries (archives). Linkers treat archives differently from object files: all object files participate in linking, while archives will only participate in linking if they can satisfy at least one undefined reference (version scripts don't count). This means that in an application that does not use `use_rt!`, `libriscv_rt*.rlib` does not participate in linking at all. This behavior also causes `#[no_mangle]` and `#[used]` items to be ignored by the linker (`KEEP` in linker scripts can't keep them either), leading to a long-standing bug in Rust ([rust-lang/rust#47384][2]). The situation changed with the merge of [rust-lang/rust#95604][1]. To fix [rust-lang/rust#47384][2], this PR introduced a synthetic object file containing references to all symbols pertaining to `#[no_mangle]` and `#[used]` items. `libriscv_rt*.rlib` now participates in linking, but the expectation is that `--gc-sections` will remove unused items in the end, unless they are explicitly retained by `KEEP` in linker scripts or other means. This change unexpectedly caused breakage in the tests for `qemu_sifive_u _s_rv(64,32)` targets, which use a custom startup routine instead of `use_rt!`. For some reason, the linker didn't respect `--gc-sections` for some items from `libriscv_rt*.rlib` and decided to include them in the output binary. These items reference symbols that are defined by the `riscv-rt` linker script, which isn't used by `qemu_sifive_u_s_ rv(64,32)` targets, hence the linker error. The thing is, `riscv-rt` outputs these items in the `.init` section[3], which is one of the section names hard-coded in LLD[4] to be excluded from `--gc-sections`. As a result, these items were considered for inclusion despite being referenced from nowhere. This commit works around this problem by making `r3_port_riscv`'s `riscv-rt` dependency optional, ensuring that `riscv-rt` participates in linking only if needed by the application. [1]: rust-lang/rust#95604 [2]: rust-lang/rust#47384 [3]: https://github.com/rust-embedded/riscv-rt/blob/7de3d2744a465ad723519c04f13c56664e138cb9/asm.S#L20 [4]: https://github.com/llvm/llvm-project/blob/b86440ecde5c1dec5b898a3f1bc08ab9853d5ed9/lld/ELF/MarkLive.cpp#L183
Fixes linker errors in an application that is linked to `r3_port_riscv` and does not use the `riscv-rt`-compatible startup routine provided by `r3_port_riscv::use_rt!`. Rust implements `rlib`s as static libraries (archives). Linkers treat archives differently from object files: all object files participate in linking, while archives will only participate in linking if they can satisfy at least one undefined reference (version scripts don't count). This means that in an application that does not use `use_rt!`, `libriscv_rt*.rlib` does not participate in linking at all. This behavior also causes `#[no_mangle]` and `#[used]` items to be ignored by the linker (`KEEP` in linker scripts can't keep them either), leading to a long-standing bug in Rust ([rust-lang/rust#47384][2]). The situation changed with the merge of [rust-lang/rust#95604][1]. To fix [rust-lang/rust#47384][2], this PR introduced a synthetic object file containing references to all symbols pertaining to `#[no_mangle]` and `#[used]` items. `libriscv_rt*.rlib` now participates in linking, but the expectation is that `--gc-sections` will remove unused items in the end, unless they are explicitly retained by `KEEP` in linker scripts or other means. This change unexpectedly caused breakage in the tests for `qemu_sifive_u _s_rv(64,32)` targets, which use a custom startup routine instead of `use_rt!`. For some reason, the linker didn't respect `--gc-sections` for some items from `libriscv_rt*.rlib` and decided to include them in the output binary. These items reference symbols that are defined by the `riscv-rt` linker script, which isn't used by `qemu_sifive_u_s_ rv(64,32)` targets, hence the linker error. The thing is, `riscv-rt` outputs these items in the `.init` section[3], which is one of the section names hard-coded in LLD[4] to be excluded from `--gc-sections`. As a result, these items were considered for inclusion despite being referenced from nowhere. This commit works around this problem by making `r3_port_riscv`'s `riscv-rt` dependency optional, ensuring that `riscv-rt` participates in linking only if needed by the application. P.S. An exclamation mark (`!`) in the commit headline will be used to indicate breaking changes from now on, as per [Conventional Commits 1.0.0][5]. [1]: rust-lang/rust#95604 [2]: rust-lang/rust#47384 [3]: https://github.com/rust-embedded/riscv-rt/blob/7de3d2744a465ad723519c04f13c56664e138cb9/asm.S#L20 [4]: https://github.com/llvm/llvm-project/blob/b86440ecde5c1dec5b898a3f1bc08ab9853d5ed9/lld/ELF/MarkLive.cpp#L183 [5]: https://www.conventionalcommits.org/en/v1.0.0/
Fixes linker errors in an application that is linked to `r3_port_riscv` and does not use the `riscv-rt`-compatible startup routine provided by `r3_port_riscv::use_rt!`. Rust implements `rlib`s as static libraries (archives). Linkers treat archives differently from object files: all object files participate in linking, while archives will only participate in linking if they can satisfy at least one undefined reference (version scripts don't count). This means that in an application that does not use `use_rt!`, `libriscv_rt*.rlib` does not participate in linking at all. This behavior also causes `#[no_mangle]` and `#[used]` items to be ignored by the linker (`KEEP` in linker scripts can't keep them either), leading to a long-standing bug in Rust ([rust-lang/rust#47384][2]). The situation changed with the merge of [rust-lang/rust#95604][1]. To fix [rust-lang/rust#47384][2], this PR introduced a synthetic object file containing references to all symbols pertaining to `#[no_mangle]` and `#[used]` items. `libriscv_rt*.rlib` now participates in linking, but the expectation is that `--gc-sections` will remove unused items in the end, unless they are explicitly retained by `KEEP` in linker scripts or other means. This change unexpectedly caused breakage in the tests for `qemu_sifive_u _s_rv(64,32)` targets, which use a custom startup routine instead of `use_rt!`. For some reason, the linker didn't respect `--gc-sections` for some items from `libriscv_rt*.rlib` and decided to include them in the output binary. These items reference symbols that are defined by the `riscv-rt` linker script, which isn't used by `qemu_sifive_u_s_ rv(64,32)` targets, hence the linker error. The thing is, `riscv-rt` outputs these items in the `.init` section[3], which is one of the section names hard-coded in LLD[4] to be excluded from `--gc-sections`. As a result, these items were considered for inclusion despite being referenced from nowhere. This commit works around this problem by making `r3_port_riscv`'s `riscv-rt` dependency optional, ensuring that `riscv-rt` participates in linking only if needed by the application. P.S. I'll try to use an exclamation mark (`!`) in the commit headline to indicate breaking changes from now on, as per [Conventional Commits 1.0.0][5]. [1]: rust-lang/rust#95604 [2]: rust-lang/rust#47384 [3]: https://github.com/rust-embedded/riscv-rt/blob/7de3d2744a465ad723519c04f13c56664e138cb9/asm.S#L20 [4]: https://github.com/llvm/llvm-project/blob/b86440ecde5c1dec5b898a3f1bc08ab9853d5ed9/lld/ELF/MarkLive.cpp#L183 [5]: https://www.conventionalcommits.org/en/v1.0.0/
PR rust-lang#95604 introduced a "synthetic object file to ensure all exported and used symbols participate in the linking". One constraint on this file is that for MIPS-based targets, its architecture-specific ELF flags must be the same as all other object files passed to the linker. That's enforced by LLD, here: https://github.com/llvm/llvm-project/blob/llvmorg-16.0.6/lld/ELF/Arch/MipsArchTree.cpp#L77 The current approach to determining e_flags for 32-bit was implemented in PR rust-lang#96930, which links to this issue that summarizes the problem well: ayrtonm/psx-sdk-rs#9 > ... the temporary object file is created with an e_flags which is > invalid for 32-bit MIPS targets. The main issue is that it omits the ABI > bits (EF_MIPS_ABI_O32) which implies it uses the N64 ABI. To enable the N32 MIPS ABI (which succeeded O32), this patch enables setting the synthetic object's ABI based on the target "llvm-abiname" field, if it's given; otherwise, the O32 ABI is assumed for 32-bit MIPS targets. More information about the N32 ABI can be found here: https://web.archive.org/web/20160121005457/http://techpubs.sgi.com/library/manuals/2000/007-2816-005/pdf/007-2816-005.pdf
Support explicit 32-bit MIPS ABI for the synthetic object PR rust-lang#95604 introduced a "synthetic object file to ensure all exported and used symbols participate in the linking". One constraint on this file is that for MIPS-based targets, its architecture-specific ELF flags must be the same as all other object files passed to the linker. That's enforced by LLD, here: https://github.com/llvm/llvm-project/blob/llvmorg-16.0.6/lld/ELF/Arch/MipsArchTree.cpp#L77 The current approach to determining e_flags for 32-bit was implemented in PR rust-lang#96930, which links to this issue that summarizes the problem well: ayrtonm/psx-sdk-rs#9 > ... the temporary object file is created with an e_flags which is > invalid for 32-bit MIPS targets. The main issue is that it omits the ABI > bits (EF_MIPS_ABI_O32) which implies it uses the N64 ABI. To enable the N32 MIPS ABI (which succeeded O32), this patch enables setting the synthetic object's ABI based on the target "llvm-abiname" field, if it's given; otherwise, the O32 ABI is assumed for 32-bit MIPS targets. More information about the N32 ABI can be found here: https://web.archive.org/web/20160121005457/http://techpubs.sgi.com/library/manuals/2000/007-2816-005/pdf/007-2816-005.pdf
Fix #50007 and #47384
This is the synthetic object file approach that I described in #95363 (comment), allowing all exported and used symbols to be linked while still allowing them to be GCed.
Related #93791, #95363
r? @petrochenkov
cc @carbotaniuman