Are there any compatibility guarantees between the cxxbridge executable and cxx macros?

[doronbehar]

Hello,

TL;DR: I want to build packages that depend on cxx macros with a cxxbridge executable that might be of different version. Is that safe? Is it possible you'd be able to guarantee e.g that any cxxbridge version 1.0.x is compatible with any cxx macro 1.0.y?

Details:

I don't use crate directly, and I don't know how to write Rust. I am simply a distributor of the taskwarrior package on NixOS. taskwarrior is using corrosion, which uses cxxbridge and cxx macros (perhaps taskwarrior uses cxx macros directly as well).

Nix's build system has this bug feature that we disallow network usage during builds. All code from the internet has to prove that it can be fetched from the internet in a reproducible manner (we use hashes for that), and after all code is fetched, we "close" the internet connection and try to build our packages.

corrosion uses cxxbridge and cxx macros and distributes a cmake file that checks that cxxbridge --version is the same version as in the Cargo.lock of the built project (in this case, taskwarrior). The problem we experience is that taskwarrior uses a certain version of cxx in their Cargo.lock file, and if these versions are not matched, the corrosion cmake rules tries to fetch the correct cxxbridge from the internet via a plain cargo install cxx command.

Our distribution does provide a cxxbridge executable that is checked in $PATH by corrosion before spawning the (failing without internet) cargo install cxx command. But, the problem is that it is very hard for use distributors to make sure these 2 versions are the same - the version of the distributed cxx-rs Nix package which provides the cxxbridge executable, and the cxx version in taskwarrior's Cargo.lock file.

I asked corrosion to perhaps loose their version check of the cxx version from the built project's Cargo.lock file and the cxxbridge --version, but they don't want to assume any such compatibility is guaranteed. Hence I ask the question above :).

Best,

[dtolnay]

That is definitely not guaranteed to work, definitely does not work historically, and I do not believe that such a guarantee would be possible to make going forward. You need cxxbridge-cmd and cxx with identical versions. This is like serde_derive and serde which has the same requirement. If this isn't going to be possible to meet reliably for NixOS, then you will need projects packaged for NixOS to use a different tool for interop than this one.

[doronbehar]

OK, thanks. I'll need to consult the NixOS folks to find a none hacky workaround. Thanks for the reply.

[dtolnay]

The correct solution is putting your cxxbridge-cmd into Cargo.lock via an unconditionally disabled Cargo dep [target.'cfg(any())'.dependencies] cxxbridge-cmd = "1". You do not need artifact dependencies for this. Then teaching Corrosion to build and use that specific cxxbridge-cmd binary from source according to the lockfile, instead of using whatever arbitrary preinstalled one from $PATH.

This is how we do it for my work's Buck codebase. We have used dozens of cxx versions over 4 years, in a project built by many thousand developers, and never had any version mismatch issue. The cxx and cxxbridge-cmd are updated in tandem within the same lockfile.

[doronbehar]

Ah OK, thanks for the advice, it sounds right. I thought about it myself initially, but the suggestion got lost when I went deeper in the dependency chain. Let's see what @djmitche thinks about it (After GothenburgBitFactory/taskwarrior#3705).

[dtolnay]

That is more of a Corrosion issue than a Taskwarrior issue. The corrosion_add_cxxbridge function in Corrosion ignores the user's lockfile for everything except the cxx version. That needs to be fixed in Corrosion.

It should be doing something along the lines of the following to install an arbitrary bin crate from an arbitrary lockfile.

#!/bin/bash

set -euo pipefail
shopt -s extglob

if [ "$#" -ne 2 ]; then
    echo "Usage: $0 cxxbridge-cmd path/to/Cargo.lock" >&2
    exit 1
fi

CRATE=$1
LOCKFILE=$2
DIR=$(mktemp "build.${CRATE}.XXXXXX" --directory --tmpdir)
cp "${LOCKFILE}" "${DIR}/Cargo.lock"
cat >"${DIR}/Cargo.toml" <<MANIFEST
    [package]
    name = "build"
    resolver = "2"
    [target.'cfg(any())'.dependencies]
    ${CRATE} = "*"
MANIFEST
mkdir "${DIR}/src"
touch "${DIR}/src/lib.rs"
cargo vendor "${DIR}/vendor" --quiet --versioned-dirs --offline --manifest-path "${DIR}/Cargo.toml"
VENDOR=("${DIR}/vendor/${CRATE}-"+([0-9])"."*)
cp "${LOCKFILE}" "${VENDOR}/Cargo.lock"
cargo build --quiet --release --offline --manifest-path "${VENDOR}/Cargo.toml"
echo "${VENDOR}/target/release"

[doronbehar]

Thanks for the script @dtolnay, I agree that corrosion should only use the cxxbridge-cmd crate from the including project's Cargo.lock, I'll cc @jschwe per:

• Loose version check of cxxbridge executable? corrosion-rs/corrosion#582
• Is it not possible to tell corrosion to use pre-fetched crates and NOT attempt to download? corrosion-rs/corrosion#578

In my case, due to the Nix requirement that all crates should be prefetched, I still think that when taskwarrior will actually call the cargo build command you wrote above on the vendor directory, the cxxbridge-cmd will be included there. Currently, cxxbridge-cmd is not a dependency in taskwarrior's Cargo.toml:

https://github.com/GothenburgBitFactory/taskwarrior/blob/e5ab1bc7a5f5a1397f41dc7609f0d5d7ae73a441/src/taskchampion-cpp/Cargo.toml#L10-L12

[dtolnay]

Let's continue this in a Corrosion issue.