Cross-compilation from LInux to x86_64-apple-darwin fails due to bad "cc" command line params

[John-Nagle]

Rust compilation on Ubuntu 22.04.2 LTS with target x86_64-apple-darwin fails with a build error:

cc: error: unrecognized command-line option '-arch'
cargo generated a cc command that can't work.

cc is (Ubuntu 11.3.0-1ubuntu1-22.04.1) 11.3.0, installed with Ubuntu 22.04.2 LTS.

Reproduce by:

rustup target add x86_64-apple-darwin
cargo new hello_world
cd hello_world
cargo build 

Expected a successful cross-compile.

Meta

rustc --version --verbose:

rustc 1.69.0 (84c898d65 2023-04-16)
binary: rustc
commit-hash: 84c898d65adf2f39a5a98507f1fe0ce10a2b8dbc
commit-date: 2023-04-16
host: x86_64-unknown-linux-gnu
release: 1.69.0
LLVM version: 15.0.7

Discussion

It's clear why this doesn't work. Cargo/rustc is invoking the platform's linker in a cross-compile, and the default linker is not capable of that cross-platform action. So this is a tool dependency problem. "rustup" installed support for that target with no errors, and if dependency checks needed to be made, they were not made at that time.

Note that the hard work is done. The compile is succesful. It's only the linking that does not work.

It can be argued that this particular cross-compilation pair is not supported. However, both the compiling platform and the target platform are Rust Tier 1 supported targets. "Tier 1 targets can be thought of as 'guaranteed to work'." There are no footnotes to the target table indicating that this source->target combination does not work.

Policy on target support says that "Tier 2 targets should, if at all possible, support cross-compiling. Tier 2 targets should not require using the target as the host for builds, even if the target supports host tools." That's for Tier 2. This is a Tier 1 target. For Tier 1 with Host Tools, the policy notes "Providing host tools does not exempt a target from requirements to support cross-compilation if at all possible."

So, if this cross-compile is technically possible, it is a bug that it does not work.

It is technically possible, since at least two unsupported workarounds exist.

Workarounds

There are at least two unsupported workarounds for this, using different tool chains.

1. Osxcross This is a set of scripts that collects tools capable of this build. This approach may require the use of proprietary Apple, Inc. libraries, which is inconsistent with Rust policy.
2. Zig Using components of the Zig toolchain appears to work. This approach does not seem to require any proprietary Apple, Inc. code.

Industry trends

Cross-platform support is becoming more widespread. Microsoft recently announced that their Windows OS is now fully supported on some Apple hardware. Apple is now offering a game porting toolkit for converting Windows games to Apple targets. The "walled garden" model seems to be weakening, at least for desktops. Zig, a language which competes in Rust's space, already supports this cross-compile.

So this cross-compile should be fully supported. Thank you.

[Jules-Bertholet]

@rustbot label A-cross O-linux O-macos

[jyn514]

The compiler appears to be unconditionally using the linker-flavor of the target instead of the flavor of the host:

rust/compiler/rustc_target/src/spec/apple_base.rs

Lines 101 to 104 in b80e0b7

	let mut args = TargetOptions::link_args(
	LinkerFlavor::Darwin(Cc::No, Lld::No),
	&["-arch", arch, "-platform_version"],
	);

[jyn514]

ah, this won't work anyway because the macOS SDK needs to be installed: https://github.com/tpoechtrager/osxcross#how-does-it-work

[John-Nagle]

The Zig compiler people somehow got past that.

[workingjubilee]

Sure, let's just ask @andrewrk if this was before or after the custom linker or if it was in fact overcome as opposed to merely avoided.

[workingjubilee]

In any case, the platform support policy has only been enforced strictly for new targets. For "grandfathered" targets like x86_64-apple-darwin, there are de facto deviations and failures.

This is much more obvious for x86_64-pc-windows-msvc, which is Windows 7, but it is not in fact tested to the standard of a tier 1 platform, i.e. "tests pass on every commit". It's not much of a test if it doesn't test the actual version, and CI tests on Windows 10 as those are the CI images we can actually obtain. "Creatively" interpreting the policy is considered more acceptable than suddenly pulling the plug on everyone who still wants to ship Rust binaries to Windows 7 and Windows 8. It is explicitly footnoted, but that does not change a policy deviation being a deviation. Likewise there is no formal maintainer team for most of the current batch of tier 1 targets.

Anyways, it seems that the Zig cross-compilation story is not magic or bulletproof:

• ability to link against darwin frameworks (such as CoreFoundation) when cross compiling ziglang/zig#1349
• use case: cross-compiling to macOS with a dependency on iconv ziglang/zig#10485
• cross compile to x86_64-macos get error: "ld64.lld: error: Unable to find library for -liconv" ziglang/zig#9725
• zig c++: Unwinding across shared library boundaries doesn't work ziglang/zig#11422
• x86_64-macos incorrectly includes macOS libs/headers when cross compiling to Linux ziglang/zig#11066

Which I am not going to cast aspersions on! It merely shows an immediate problem for us: Rust is much more pedantic about UTF-8 than Zig is and it is... very beneficial to our binary size if we can use the system libraries. Merely having more struggles linking libiconv seems like a bad start. "You can cross-compile from Linux or Windows to macOS but it doubles your starting binary size compared to a native build" sounds plausible as something we might want to offer, but also would be a Big Oof if that is the tradeoff we would have to make.

Presumably this issue can still be solved rather than "fixed", but it seems effortlessly cross-compiling a programming language with a different set of features than Zig has to macOS is not an already-solved problem with no tradeoffs.

[andrewrk]

For iconv, you should be able to solve it the same way as libSystem, which is to ship the tbd files with the Rust toolchain. These can be given to the linker to produce dynamic dependencies on the target system's iconv shared libraries. Note that Zig does not provide Unicode APIs in its standard library, nor does the language depend on Unicode, so iconv is simply not a dependency of most Zig projects.

For frameworks, our plan is to rely on third party packages that provide them rather than shipping them directly with zig. The Rust equivalent I believe would be a Cargo crate for each framework.

The headers/libs things looks like a silly bug that I'm not sure how it hasn't been fixed yet, and the unwinding thing looks like a nightmare level linker bug that we'll unfortunately have to address eventually.

[workingjubilee]

Wonderful! Thank you for the info, @andrewrk, that will make improving the situation here much easier. The "framework crate" answer sounds pretty close to how things work-out in practice with the ecosystem's convention of Rust {lib}-sys crates.

Also, I have been debriefed more on the background issues for libiconv:

• Don't link against iconv on apple targets when used by std libc#2944
• libiconv is still unconditionally linked to all binaries on macOS libc#2870 (comment)

So it looks like we don't need a dependency on libiconv: Rust provides its own UTF-8 handling, so we aren't even benefiting from libiconv for binary size! It's purely an accident! Kind of annoying.

[tindzk]

Besides Zig, you could try rust-lld which is bundled with the Rust toolchain. I am using these commands in GitHub Actions (ubuntu-latest) to build for macOS:

curl -L https://github.com/roblabla/MacOSX-SDKs/releases/download/13.3/MacOSX13.3.sdk.tar.xz | tar xJ
export SDKROOT=$(pwd)/MacOSX13.3.sdk/
export PATH=$PATH:~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/
export CARGO_TARGET_X86_64_APPLE_DARWIN_LINKER=rust-lld
rustup target add x86_64-apple-darwin
cargo build --release --target x86_64-apple-darwin

[csaben]

Besides Zig, you could try rust-lld which is bundled with the Rust toolchain. I am using these commands in GitHub Actions (ubuntu-latest) to build for macOS:

curl -L https://github.com/roblabla/MacOSX-SDKs/releases/download/13.3/MacOSX13.3.sdk.tar.xz | tar xJ
export SDKROOT=$(pwd)/MacOSX13.3.sdk/
export PATH=$PATH:~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/
export CARGO_TARGET_X86_64_APPLE_DARWIN_LINKER=rust-lld
rustup target add x86_64-apple-darwin
cargo build --release --target x86_64-apple-darwin

you are a legend.

[John-Nagle]

I'll have to try that.

It turns out that my ui-mock program, which exercises the Egui/Rend3/WPGU/Winit stack, ran properly on MacOS when built by someone who had a Mac. On Apple silicon, even. No Mac-related mods required. So I'll have to try a cross-compile.

Portability is working better than expected. I build the Windows version on Linux and release it that way.

[drHyperion451]

Besides Zig, you could try rust-lld which is bundled with the Rust toolchain. I am using these commands in GitHub Actions (ubuntu-latest) to build for macOS:

curl -L https://github.com/roblabla/MacOSX-SDKs/releases/download/13.3/MacOSX13.3.sdk.tar.xz | tar xJ
export SDKROOT=$(pwd)/MacOSX13.3.sdk/
export PATH=$PATH:~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/
export CARGO_TARGET_X86_64_APPLE_DARWIN_LINKER=rust-lld
rustup target add x86_64-apple-darwin
cargo build --release --target x86_64-apple-darwin

I just used export CARGO_TARGET_X86_64_APPLE_DARWIN_LINKER=rust-lld on my mac and it worked perfectly

[charro]

Besides Zig, you could try rust-lld which is bundled with the Rust toolchain. I am using these commands in GitHub Actions (ubuntu-latest) to build for macOS:

curl -L https://github.com/roblabla/MacOSX-SDKs/releases/download/13.3/MacOSX13.3.sdk.tar.xz | tar xJ
export SDKROOT=$(pwd)/MacOSX13.3.sdk/
export PATH=$PATH:~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/
export CARGO_TARGET_X86_64_APPLE_DARWIN_LINKER=rust-lld
rustup target add x86_64-apple-darwin
cargo build --release --target x86_64-apple-darwin

This is working great, thanks for sharing!
My 2 cents: If you want to use directly rust Docker image rust:latest in your CI instead of ubuntu, I'm doing so adding what's missing for the cross-build:

apt-get update -yqq
apt-get install -yqq --no-install-recommends build-essential
apt-get install -yqq mingw-w64   # For Windows build
apt-get install -yqq clang gcc g++ zlib1g-dev libmpc-dev libmpfr-dev libgmp-dev cmake libxml2-dev libssl-dev zip unzip

[piaoger]

Above method really works. But in my case which may use some c/c++ deps, there will have below errors:

  ...
  cargo:warning=cc: error: unrecognized command line option ‘-arch’
  cargo:warning=cc: error: x86_64: No such file or directory
  cargo:warning=cc: error: unrecognized debug output level ‘full’
  cargo:warning=cc: error: unrecognized command line option ‘-arch’
 ...

error: unable to find framework 'CoreFoundation'. searched paths:  none

• Solution:

Just uses SDKROOT + zigbuild !!

FROM

cargo build --release --target x86_64-apple-darwin

TO

cargo zigbuild --release --target x86_64-apple-darwin 

[John-Nagle]

Oh, nice. The Zig people are putting in effort on cross-platform. I'll have to try that.

[Clownsw]

That is a great act.

[vdudouyt]

It worked like a charm for me after specifying macOS SDK path by @csaben's suggestion:

$ export CARGO_TARGET_AARCH64_APPLE_DARWIN_LINKER=/root/osxcross/target/bin/x86_64-apple-darwin23.5-cc
$ cargo build --target=aarch64-apple-darwin
   Compiling hello v0.1.0 (/root/hello)
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.13s