Cargo (rust release version) can't bootstrap itself

[lucab]

I think that this is bug I hit is the other half of #1178, but I'm an unsure, so please read through the logs.

So, imagine that you are packaging the version of cargo released with rust-1.0.0alpha, and you need to bootstrap it. From #1178, you learn that you want cargo at revision 23ed197. So you locally install rustc/cargo from 1.0.0-alpha, and prepare to build a system version of cargo with that:

$ which rustc
/usr/local/bin/rustc
$ rustc -V
rustc 1.0.0-alpha (44a287e6e 2015-01-08 17:03:40 -0800)
$ which cargo
/usr/local/bin/cargo
$ cargo -V
cargo 0.0.1-pre-nightly (23ed197 2015-01-06 02:37:42 +0000)
$ git log HEAD | head -1
commit 23ed197405509ae17654a86b6d59a8fb6d367f9e

$ ./configure --prefix=/usr --local-rust-root=/usr/local --local-cargo=/usr/local/bin/cargo
configure: 
configure: CFG_PREFIX           := /usr 
configure: CFG_LOCAL_RUST_ROOT  := /usr/local 
configure: 
configure: using rustc at: /usr/local with version: rustc 1.0.0-alpha (44a287e6e 2015-01-08 17:03:40 -0800)
configure: CFG_LOCAL_CARGO      := /usr/local/bin/cargo 
configure: leaving ./Makefile unchanged
configure: mv config.tmp config.mk
configure: 
configure: complete

Then you try to build via make, and that's what you get:

$ make
"/usr/local/bin/rustc" -V
rustc 1.0.0-alpha (44a287e6e 2015-01-08 17:03:40 -0800)
/usr/local/bin/cargo build --target x86_64-unknown-linux-gnu  
    Updating registry `https://github.com/rust-lang/crates.io-index`
    Updating git repository `https://github.com/carllerche/hamcrest-rust.git`
 Downloading git2 v0.1.7
 Downloading log v0.1.7
 Downloading curl-sys v0.1.3
 Downloading flate2 v0.1.3
 Downloading pkg-config v0.1.3
 Downloading term v0.1.2
 Downloading gcc v0.1.3
 Downloading rustc-serialize v0.2.4
 Downloading toml v0.1.8
 Downloading regex v0.1.8
 Downloading openssl-sys v0.2.12
 Downloading docopt v0.6.23
 Downloading tar v0.1.5
 Downloading libz-sys v0.1.0
 Downloading libssh2-sys v0.1.1
 Downloading time v0.1.9
 Downloading libgit2-sys v0.1.5
 Downloading semver v0.1.9
 Downloading curl v0.1.7
 Downloading glob v0.1.4
 Downloading url v0.2.13
 Downloading miniz-sys v0.1.1
   Compiling gcc v0.1.3
   Compiling pkg-config v0.1.3
   Compiling glob v0.1.4
   Compiling semver v0.1.9
/home/lbruno/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/semver-0.1.9/src/version.rs:174:65: 174:66 error: obsolete syntax: `|uint| -> bool` closure type syntax
/home/lbruno/.cargo/registry/src/github.aaakk.us.kg-1ecc6299db9ec823/semver-0.1.9/src/version.rs:174 fn take_nonempty_prefix<T:Iterator<Item=char>>(rdr: &mut T, pred: |char| -> bool)
                                                                                                                                                             ^
note: use unboxed closures instead, no type annotation needed
[many more error pages for all the other crates follow here]

I was puzzled for a brief moment by this, but then I think I started understanding what's going on here.

Cargo, like rustc, has a circular dependency on itself. Additionally to that, cargo has two external dependencies: rustc and several crates. Obviously, those crates have an implicit dependencies on some specific rustc (nightly) version, due to syntax/libs evolution.

As such, cargo and those crates are locked to a specific nightly version. It may happen (and I think this is the case here) that breaking changes exist between that nightly and the released alpha/beta/stable. In that case, cargo and crates will build with the nightly, but fail with the alpha/beta/stable.

I think you'll never experience this, as your cargo and rust CI toolchains are tightly locked and just download the nightly versions they want (or, if you do, you just bump the snapshots). However, from a distribution POV, this means that in order to simply locally build a released version of rustc+cargo, I have to go digging through rust+cargo history, fetch several stage0 and nightlies, build at least two different copies of rust and hope for the best.

This is my first encounter with cargo packaging (RIP rustpkg) so I hope that my analysis is wrong somewhere. And I'm still unsure how to proceed from here on.

[lucab]

OTOH, it looks like current master builds with 1.0.0-alpha because all the crates have been updated and some code in cargo itself amended. I think this is mostly by chance, as no other breaking changes occurred in the middle. I could just package this (once I have a clearer plan on how to handle all the dep-crates) and call it a day.

However in the long term this seems a bit unfortunate, as I'll be always providing a newer-than-released cargo instead of the same version you ship in rust installer.

I think that, if you don't plan/want to change how your (upstream) packaging machinery works, it would be great to at least fork off a branch at the same time of the release and cherry-pick there the Cargo.toml updates and minor code fixes needed to build with the corresponding rustc release. I think that this may simplify (downstream) packaging quite a bit.

[ghost]

I'm not sure how relevant this is but I want to mention that latest cargo master builds with this:

"/home/emacs/build/cargo/rustc/bin/rustc" -V
rustc 1.0.0-nightly (4be79d6ac 2015-01-23 16:08:14 +0000)
"target/snapshot/bin/cargo" --version
cargo 0.0.1-pre-nightly (bb28e71 2015-01-22 06:06:34 +0000)

^ that is, after you run .travis.install.deps.sh and assuming you compile it with:

./configure --local-rust-root="$PWD/rustc"
make

[lucab]

@emanueLczirai yes, see also my comment above. The report is mostly about currently released version (as shipped in -alpha installer) and how to get the next (-beta) release easier to package.

[alexcrichton]

This is largely intended as-is today as Cargo pins itself to a particular Cargo snapshot (like rustc) as well as a particular rustc version. It is intended that Cargo builds with entirely stable code at the 1.0 release of Rust at which point I believe that the necessity of a pinned compiler (or a pinned Cargo) will become much less pressing.

For now though the two are in enough flux that I think we need to continue the current strategy for the near future though. As a data point though I didn't have to update the Cargo snapshot for a month and I have a feeling that will only get longer (rustc is another story though!).

[lucab]

Sure, understandable. The point of the report was for downstream packager to be aware of this issue and for you to avoid it for 1.0.0 stable (which is the whole reasons of having beta releases). Otherwise we may end up with a stable cargo which is a hell to sanely build downstream :)

[alexcrichton]

Yeah definitely, I'm certainly quite keen to fix issues like this! I'd definitely be willing to touch base closer to when Rust itself is stable to make sure Cargo is easily package-able