bootstrap tries to download the wrong link from ci-artifacts.rust-lang.org if the local repository is shallow

[TheWastl]

Steps to reproduce

git clone --depth 1 https://github.com/rust-lang/rust.git
cd rust
./x.py setup compiler
./x.py

Expected output

An error message indicating that a full (or at least longer) git history is required.

Actual output (of final x.py invocation)

info: Downloading and building bootstrap before processing --help
      command. See src/bootstrap/README.md for help with common
      commands.
Updating only changed submodules
Submodules updated in 0.01 seconds
downloading https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
curl: (22) The requested URL returned error: 404 


spurious failure, trying again
downloading https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
curl: (22) The requested URL returned error: 404 


spurious failure, trying again
downloading https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
curl: (22) The requested URL returned error: 404 


spurious failure, trying again
downloading https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
curl: (22) The requested URL returned error: 404 


spurious failure, trying again
downloading https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz
curl: (22) The requested URL returned error: 404 

failed to run: curl -# -y 30 -Y 10 --connect-timeout 30 --retry 3 -Sf -o /tmp/tmpt60y4od2 https://ci-artifacts.rust-lang.org/rustc-builds//rust-dev-nightly-x86_64-unknown-linux-gnu.tar.xz

The correct link would include a commit hash inbetween the //. But the required commit hash is not available because there is no full history.

[TheWastl]

Hm, can I do this?

@rustbot label +A-rustbuild

[jyn514]

Mentoring instructions: find what this command outputs when the repository is shallow:

rust/src/bootstrap/bootstrap.py

Lines 669 to 672 in ae90dcf

	merge_base = [
	"git", "rev-list", "[email protected]", "-n1",
	"--merges", "--first-parent", "HEAD"
	]

If it's an error, see if you can check for a shallow repository somehow and give a hard error instead of continuing onward just to fail.

[jyn514]

Note that this shouldn't need to the full history of the repo, just the past few days, since PRs are constantly being merged.

[Mark-Simulacrum]

We should only need one (bors merge) commit, I believe - each bors merge has the full set of artifacts.

[ghost]

The git rev-list [email protected] -n1 --merges --first-parent HEAD command prints nothing in my shallow clone. There is a bors merge commit in my shallow clone though:

$ git log --oneline
859d3b67 (HEAD -> no-mangle-method-wip) Ignore `#[no_mangle]` and `#[export_name = "..."]` on generic functions and trait methods
a563b825 Adjust `check_no_mangle` and `check_export_name` to warn/error on `#[no_mangle]`/`#[export_name]` on trait methods
0bb2ea65 Adjust `#[no_mangle]`-related checks and lints for `impl` items
c84beefd Add associated functions that have custom linkage to `reachable_set`
eb2226b1 (grafted, origin/master, master) Auto merge of #85296 - bjorn3:plugin_cleanup, r=petrochenkov

The git log command before #87532 used to work for me -- I was able to enable download-ci-llvm in my shallow clone.

I tried to fetch more commits, and then git rev-list [email protected] -n1 --merges --first-parent HEAD started to work:

$ git rev-list [email protected] -n1 --merges --first-parent HEAD
eb2226b1f174f3cc644275ef8663be6295a7f704

But git rev-list [email protected] -n1 --merges --first-parent HEAD -- /somewhere/src/llvm-project /somewhere/src/bootstrap/download-ci-llvm-stamp /somewhere/src/version still prints nothing.

[Mark-Simulacrum]

cc @tlyu

I would have expected the git log and git rev-list to be 'identical', in some sense, so perhaps that's something worth filing upstream as well. But regardless, I am inclined to revert #87532 to fix this issue; it was primarily a 'hopefully more robust' solution and to my knowledge wasn't actually fixing any known bugs, so since it caused a bug seems good to revert it for now...

[tlyu]

I think it's possible that the shallow clone is causing the --merges predicate to fail. Note the grafted decoration in

eb2226b1 (grafted, origin/master, master) Auto merge of #85296 - bjorn3:plugin_cleanup, r=petrochenkov

The git documentation implies that shallow clones use something akin to the info/grafts (fake parentage for commits) facility for shallow clones. Maybe removing --merges is sufficient? I'm a bit wary because bors seems to create non-merge commits on occasion (for squashes), but maybe it never does so on the first-parent history.

I haven't had a chance to poke at a shallow clone in detail yet.

[BGR360]

Would love to see movement on this. I don't want to allocate ~20GB of laptop space to a deep clone of the repository in order to simply be able to build the compiler.

When i was messing around, removing "--merges" as @tlyu suggested did the trick.

[jyn514]

@BGR360 feel free to make a PR with that change :)

[BGR360]

@jyn514 I wasn't sure if there was some reason not to do that:

I'm a bit wary because bors seems to create non-merge commits on occasion (for squashes), but maybe it never does so on the first-parent history.

I don't really grok most of what @tlyu said there, but if you think that's a safe change, then sure thing!

[Mark-Simulacrum]

As long as we keep first parent, dropping merges should be ok, any commit in the first parent chain should have artifacts (if it's recent enough).

[tlyu]

I'm seeing a similar problem for git worktrees, not just shallow clones

I would appreciate knowing details of what behavior you're seeing with git worktrees, if you're willing to provide them.

[jyn514]

@tlyu I don't think we need to exhaustively catalogue the issues - even if removing --merges doesn't fix all the issues it will fix some of them, and we can make more improvements in follow-ups.

[tlyu]

@rustbot claim

[tlyu]

Someone should probably file a bug about git rev-list --merges failing to see merge commits in shallow clones. I don't feel like I understand the issue well enough to do so.