error: invalid instruction cargo:rustc-cdylib-link-arg from build script of pyo3 v0.14.1

[Hywan]

🐛 Bug Reports

I try to compile wasmer-python with pyo3 0.14.1. It uses pyo3 to create a Python extension with Rust. Whe compiling, I'm facing this error:

error: invalid instruction `cargo:rustc-cdylib-link-arg` from build script of `pyo3 v0.14.1`
The package pyo3 v0.14.1 does not have a cdylib target.

🌍 Environment

• macOS 11.1 aarch64
• Python 3.9.4 from Homebrew
• Rust 1.55 nightly
• PyO3 0.14.1

💥 Reproducing

When compiling examples/word-count/ with cargo build --release, we get a warning:

warning: cargo:rustc-cdylib-link-arg was specified in the build script of pyo3 v0.14.1 (/Users/hwhost/Development/pyo3/pyo3), but that package does not contain a cdylib target

Allowing this was an unintended change in the 1.50 release, and may become an error in the future. For more information, see <https://github.com/rust-lang/cargo/issues/9562>.
warning: cargo:rustc-cdylib-link-arg was specified in the build script of pyo3 v0.14.1 (/Users/hwhost/Development/pyo3/pyo3), but that package does not contain a cdylib target

Allowing this was an unintended change in the 1.50 release, and may become an error in the future. For more information, see <https://github.com/rust-lang/cargo/issues/9562>.

In my project (wasmer-python), it's an error.

[Hywan]

To conclude on a more positive aspect: PyO3 is awesome, and you're all doing an amazing job!

[davidhewitt]

Ah, that's a shame. This is #1539, but I guess that rust-lang/cargo#9562 is saying that was an unintended change.

This probably has implications for #1123, which is using similar cargo flags.

I wonder if we can provide some hooks in the pyo3-build-config crate which work around this.

@Hywan do I understand correctly that this is a blocking error for your attempting to upgrade? If so, we should probably figure out an appropriate patch fix...

[Hywan]

Yeah it's an error that forbids me to update.

I know it comes from the build.rs script where you check only against macos without checking the architecture. The linker arguments are only required for x86_64-darwin-apple, they are not required for aarch64-darwin-apple. I didn't want to put you on the wrong path in my description, but that was my feeling. Also, it's great to handle this in the build.rs script rather than in .cargo/config for sure.

[Hywan]

#1123 seems a good PR to fix the problem. Let me know, I can try anything to help you :-).

[davidhewitt]

The linker arguments are only required for x86_64-darwin-apple, they are not required for aarch64-darwin-apple

Are you sure about this? Our experience in #1330 seemed to suggest otherwise...

[Hywan]

Prior to pyo3 0.14, I had a .cargo/config file only for x86_64-apple-darwin, nothing for aarch64-apple-darwin and everything was working like a charm.

Don't ask me why though, because it's counter-intuitive.

[davidhewitt]

Could it be related to building two separate targets versus a universal binary?

[Hywan]

I've no idea. Maybe the user was running Python behind Rosetta?

[Hywan]

Any news on that front :-)?

[davidhewitt]

No progress yet, sorry. I'm kinda unsure what to do. I guess the reality is that we can't automatically apply the macOS build configuration, so we might want to add an API along the lines of pyo3_build_config::add_macos_configuration() which checks the target and adds the flags if running on macOS. Users would have to add this in their own build script, but at least it's better than requiring cargo configs?

[Hywan]

I think it's better than an error, so yeah :-)

[davidhewitt]

Just to keep everyone in the loop, as reverting this will involve writing a fair bit of new documentation, I'm planning to do this as part of a big update to the README at the weekend.

[davidhewitt]

FYI PyO3 0.14.2 is planned to go live tomorrow. #1760

[Hywan]

Thanks!

[Hywan]

I've updated to the latest Rust nighly (rustc 1.56.0-nightly (ccffcafd5 2021-08-11)), and pyo3_build_config::add_extension_module_link_args() is working on macOS M1.

Note: It doesn't mean it was a bug in rustc. I didn't dig into that. But, eh, it works!