cross picking up wrong target options in cargos config.toml

[oliverbestmann]

I've added a [target.target.x86_64-unknown-linux-gnu] section to my $HOME/.cargo/config.toml. If i am now using cross to compile for a different target, say cross build --target aarch64-unknown-linux-gnu, the build picks up the config for x86_64-unknown-linux-gnu.

In my case, i've configured

oliver@machine ~> /bin/cat ~/.cargo/config.toml
[target.x86_64-unknown-linux-gnu]
linker = "/usr/bin/clang"
rustflags = [
  "-C", "target-cpu=native",
  "-C", "link-arg=-fuse-ld=/home/oliver/.local/bin/mold"
]

And the build fails with

oliver@machine ~> cross build --target aarch64-unknown-linux-gnu --release
info: syncing channel updates for '1.56.0-x86_64-unknown-linux-gnu'
  1.56.0-x86_64-unknown-linux-gnu unchanged - rustc 1.56.0 (09c42c458 2021-10-18)
error: linker `/usr/bin/clang` not found
  |
  = note: No such file or directory (os error 2)

doesn't work with rust 1.57.0 either.
cross version 0.2.1

[Emilgardis]

We shouldn't currently use that at all, instead you'd have to place the config in .cargo/config.tomlin the manifest root if you want it to go over to the cross builder

perhaps we should support this automatically, but currently you should be able to workaround it by mounting the .cargo folder inside the builder with https://github.com/rust-embedded/cross#mounting-volumes-into-the-build-environment to any of the supported paths

[Emilgardis]

Note that that feature is currently unreleased, you'd have to install from git

cargo install cross --git https://github.com/rust-embedded/cross

[oliverbestmann]

Thing is, I don't want cross to pick that file up at all. It currently contains only configuration related to my host machine and target. But when compiling with cross on x86_64 for target aarch64, it picks up the options for the x86_64 target.
Also still broken with the the current master version.

[kdar]

Thank you so much for this issue, as this was driving me up the wall why my cross compiling was failing. I'm a little confused as to why it was happening though. I was compiling a simple program, and it would compile until I added "libc" as a dependency. The failure was:

Compiling num-integer v0.1.44
error: linker `/usr/bin/clang` not found
  |
  = note: No such file or directory (os error 2)

This would happen because my ~/.cargo/config.toml contained:

[target.x86_64-unknown-linux-gnu]
linker = "/usr/bin/clang"
rustflags = ["-C", "link-arg=--ld-path=/usr/sbin/mold"]

The confusing thing to me is that I'm targetting aarch64-linux-android. Maybe I'm misunderstanding exactly what cross does, but why does it care about the x86_64-unknown-linux-gnu target when I'm compiling for android?

Anyway, this was surprising to me here and I agree with @oliverbestmann.

[oliverbestmann]

Any update on this @Emilgardis? The issue still exists in 7599e1a. cross still tries to use the target configuration of the host system, not the actual cross target system.
It even got worse as cross now exists with return code zero, indicating no error at all.

[Emilgardis]

I'll try to get in a fix for this. Not sure exactly what is happening but have an idea how to solve it.

[sadilekivan]

Just wanted to post an issue on this exact behavior. Any progress on it?

[Emilgardis]

I'd forgotten about this, this can be solved by us setting CARGO_TARGET_<triple>_LINKER in every image we provide, since env vars take priority over toml values. However, this still feels like a bad solution.

[Alexhuszagh]

I'm working on this now, since I'm also trying to support aliases. A simple solution is the one above, while also parsing the aliases (which we would have to do anyway). The complex solution, which I want to support, is below:

However, this would require using docker exec and then cleaning up the image ourselves if we write the concatenated config.toml to the file, and copying it in an initial step to the container. Unless if you know an environment variable we can set to locate where that config file would be, so we can store it in the target directory or something.

[sadilekivan]

I tried the environment variable solution, but if the config.toml is present, it still displays the same error. I think I'm using it correctly though, because if I set a non-existing linker it shows up as another error if my config.toml is gone.

I tried export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="aarch64-linux-gnu-gcc" ; cross build --target aarch64-unknown-linux-gnu, It only works if my config.toml is not present.

[sadilekivan]

It does seem that way. For now I'm just renaming the config with a script. A pity though, an env variable fix would be less cumber some. Thanks for the fast feedback, I wasn't completely sure if that was the correct linker.

[Alexhuszagh]

Trying to find the best solution right now, and sorry for the convenience. Working on a few issues but this is pretty high on my priority list.

[Emilgardis]

I tried export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="aarch64-linux-gnu-gcc" ; cross build --target aarch64-unknown-linux-gnu, It only works if my config.toml is not present.

This is correct, however you need to forward it to the container @sadilekivan

so in Cross.toml

[target.aarch64-unknown-linux-gnu.env]
passthrough = ["CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc"]

[Alexhuszagh]

That should be handled automatically though from the latest main as of #869. This was not in v0.2.2 though. You can use the --verbose flag to confirm this.

[sadilekivan]

Yes, the flag does pass, as I mentioned earlier if I specify a non-existing linker like export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="this_isnt_a_linker". I can see the compilation fails at looking for it, so it does read/react to the variable.

so in Cross.toml

[target.aarch64-unknown-linux-gnu.env]
passthrough = ["CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc"]

I tried this, the flag gets passed too, but it still prioritizes .cargo/config.toml.

[Alexhuszagh]

Yes, the flag does pass, as I mentioned earlier if I specify a non-existing linker like export CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER="this_isnt_a_linker". I can see the compilation fails at looking for it, so it does read/react to the variable.

so in Cross.toml

[target.aarch64-unknown-linux-gnu.env]
passthrough = ["CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER=aarch64-linux-gnu-gcc"]

I tried this, the flag gets passed too, but it still prioritizes .cargo/config.toml.

Ah sorry, that's the default behavior of Cargo itself (although I believe it's [undocumented[(https://doc.rust-lang.org/cargo/reference/config.html#target), and not of cross, and therefore we need to fix this. Cargo is very inconsistent with the priority of environment variables and config settings (CARGO_INCREMENTAL overrides build.incremental, but doc.browser overrides BROWSER).

[Alexhuszagh]

So this is a breaking change no matter how we look at it: some programs likely rely on cargo's config.toml, however, this only applies to those in the project workspace or those in the cargo home. So, fully supporting all parsed versions of config.toml would break with existing behavior.

The behavior you desire, to ignore all config.toml settings, would also break with existing behavior. I've therefore set this for the 0.3.0 milestone (and am actively working on this). I aim to get this and support for aliases done for 0.3.0.

This would likely be done with the CROSS_IGNORE_CARGO_CONFIG or a similar flag. I need to think about the best way to do this, since we need the following constraints:

• It must ignore $CARGO_HOME/config and $CARGO_HOME/config.toml. This is not easy to do since we would need to rename these files, and then on program termination restore them (either via a signal interrupt or natural ending).
• It must ignore every parent of the current directory until $root/cargo/config.toml, if we are in the project subdirectory. This can be easily done by mounted anonymous volumes for each $dir/cargo subdirectory.

In case we can't restore the files (say, via a SIGKILL), we'd also need to detect if the files were temporarily removed. This wouldn't be too bad. If anyone knows a better approach, let me know. I'd love a better solution than this, since it might break cargo builds in some cases. Since it would be opt-in, however, you'd likely know that and know how to restore it.

[sadilekivan]

I'm unsure of how cross is exactly implemented with cargo, but I would expect that cross has the ability to read CLI arguments and act based upon them.

I imagine that CLI arguments should be prioritized over config variables. So that --target overwrites my option from the config, not ignoring config files settings alltogether.

[Alexhuszagh]

I'm unsure of how cross is exactly implemented with cargo, but I would expect that cross has the ability to read CLI arguments and act based upon them.

I imagine that CLI arguments should be prioritized over config variables. So that --target overwrites my option from the config, not ignoring config files settings alltogether.

The problem is we mount both CARGO_HOME and the project directory into the container, which contains the config files cargo uses. So, we're dependent on the order of how cargo interprets those environment variables unless we hide those config files from the container. We can't prevent CARGO_HOME from being mounted, but we can prevent all other config files. Unfortunately, often config settings are in CARGO_HOME since it's the preferred way to specify settings for all projects.

So the issue is not how cross interprets config settings/environment variables: we always have environment variables override those present in config files. But, this only impacts cross config options: not cargo ones.

[sadilekivan]

By playing around I tried to understand this issue and thought I'd post this description of it.

The missing linker error occurs within cargo, if building for any target (even default) as long as there is a local machine architecture target in config.toml that has a non-existing linker.

So in conclusion I think the actual issue is that cargo checks the linker path of the local target even if not called. Cargo does not do this for any other targets, unless they are called, so why the local machine one. Correct me if I'm wrong, but I don't think its necessary or maybe even desired to do that.

This happens in standalone cargo building. If cross just calls cargo (forgive the simplification) on the docker container, then it's not a cross issue at all.

[Alexhuszagh]

This happens in standalone cargo building. If cross just calls cargo (forgive the simplification) on the docker container, then it's not a cross issue at all.

It does, but cross puts cargo in an unusual environment, so I would argue this is a cross issue: users should be able to opt-in and out of global cargo config settings. This is more in the realm of a feature request, and not a bug, but I feel it's definitely something we should allow.

[sadilekivan]

So far I only tried per project .cargo/config.toml, haven't used a global one.

Edit: Nevermind the original issue did.

[Alexhuszagh]

So far I only tried per project .cargo/config.toml, haven't used a global one.

That we can very easily support (as in, skipping it). This is because the cargo subdirectory of the project won't be used for anything else (unless it's cargo's home, in which case we need to mount it), and Docker has ways of preventing subdirectories from being mounted.

To do so, you mount an anonymous data volume at the resulting path, and it prevents the data in bind mount (the volume being mounted on the host) at that subdirectory from being mounted. We actually do this already in cross, for example, to prevent the binaries in cargo home from being mounted.

[Alexhuszagh]

It's also probably worth noting that renaming the file is less iffy than it might appear: rename is guaranteed to be async-signal safe by the POSIX standard, so for any compliant POSIX-compliant UNIX operating system (and almost all UNIX-like systems, including Linux), we can guarantee unless SIGKILL is sent to the program that the config file(s) should be renamed back.

[Emilgardis]

do we need to mount with write?

[Alexhuszagh]

do we need to mount with write?

I don't think so, since we'd either need a data volume (guaranteed to be correctly but costly, and we could simply remove the config files after copying), or we can handle it outside the container (in which case, we don't need write permissions). Also, the data volume would require setup and shutdown like our remote logic which I think we should avoid as long as we can.

[Emilgardis]

I think the correct thing to do is to do this upstream in cargo. If it's not something in scope for cargo, maybe then we can consider supporting this in a "hacky" way. (I don't think renaming on host is a good solution, since then you'd probably break some simultaneous compilation where a specific config is expected to be present.

rust-lang/cargo#6728 seems like a possible fix

rust-lang/cargo#5416 seems to be the issue at hand

[Alexhuszagh]

Since both would be breaking changes, so it might take a deprecation period. But yeah, I don't want renaming on the host unless it can be done as opt-in behavior, with clear implications of what that means.

Actually, this might have been solved by rust-lang/cargo#6699, which I didn't see. I wonder... let me check (ah it's unstable).

[Emilgardis]

CARGO_INCLUDE seems like the best solution right now for nightly, that's be pretty easy to support

[Alexhuszagh]

We've got a working solution in #931 which once rebased will support both ignoring all configurations outside of CARGO_HOME (there's no good way to ignore that right now without touching the filesystem) and including all configuration files. This is pretty close to being finished. The default behavior would be the same (to avoid breaking changes), but you can specify the desired behavior via:

[build.env]
# This can be "complete", "default", or "ignore".
cargo-config = "complete"

[JakkuSakura]

We've got a working solution in #931 which once rebased will support both ignoring all configurations outside of CARGO_HOME (there's no good way to ignore that right now without touching the filesystem) and including all configuration files. This is pretty close to being finished. The default behavior would be the same (to avoid breaking changes), but you can specify the desired behavior via:

[build.env]
# This can be "complete", "default", or "ignore".
cargo-config = "complete"

Which cargo-config should I use in this case?

[sergiimk]

For users of mold here's an ugly workaround that worked for me:

Create Cross.toml file:

# This avoids cross picking up custom linker from ~/.cargo/config.toml
# See: https://github.com/cross-rs/cross/issues/621

[build.env]
passthrough = [
    "CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc",
    "RUSTFLAGS",
]

Invoke cross as:

RUSTFLAGS="" cross build --target x86_64-unknown-linux-gnu --release

[JakkuSakura]

I work around this by renaming my cargo config and renaming back when cross is finished Sincerely, Jack

…

On 5 Jan 2023 at 8:24 AM +0800, Sergii Mikhtoniuk ***@***.***>, wrote: For users of mold here's an ugly workaround that worked for me: Create Cross.toml file: # This avoids cross picking up custom linker from ~/.cargo/config.toml # See: #621 [build.env] passthrough = [ "CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc", "RUSTFLAGS", ] Invoke cross as: RUSTFLAGS="" cross build --target x86_64-unknown-linux-gnu --release — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***>

[M3t0r]

I'm on an x86_64 Arch Linux using mold, trying to cross-compile for a Raspberry Pi (aarch64-unknown-linux-gnu).

I tried #621 (comment) but didn't mange to get it to work.

The mold "How to use" section mentioned newer versions of gcc support -fuse-ld too. I had to change the value of that flag from /usr/bin/mold to just mold, but I was able to drop linker="clang" and now it works in cross as well.

My .cargo/config.toml:

[target.x86_64-unknown-linux-gnu]
rustflags = ["-C", "link-arg=-fuse-ld=mold"]

my versions

$ gcc --version
gcc (GCC) 13.2.1 20230801
Copyright (C) 2023 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ mold --version
mold 2.30.0 (c7f6a91da512ef8a2634a1bef5d4ba82104659fe; compatible with GNU ld)

$ cross --version
cross 0.2.5 (085092c 2024-03-08)
[cross] note: Falling back to `cargo` on the host.
cargo 1.77.2 (e52e36006 2024-03-26)