stdlib-git: improve integration with git-external

[vtjnash]

Previously some of the work was shifted, leading to awkward extraneous copies
and preventing usage of DEPS_GIT=1 mode (the main point of this script).
There also seemed to be unexpected failure modes in the current usage
where the files in stdlib/Pkg would routinely go missing for me. I don't know
the source of those, but I hope that this will fix it (because it makes more of
the state immutable).

Also noticed some issues with existing scripts (expanding variables at
unintended points, not rebuilding on some changes), and fixed those.

[KristofferC]

Why is this changing the Pkg sha?

[vtjnash]

It had the wrong SHA1 from what was intended here. This is now the whole commit hash, while previously it only contained the tree hash for this commit. This var is (intentionally) allowed to specify any git object (incl. branch, tag, etc.), but typically it should have the commit SHA, since that's the most general and easiest to work with (e.g. what git usually presents you).

[fredrikekre]

But why are the checksums also changed? Shouldn't the commit you set here produce the same tarball?

[vtjnash]

The name here gets inserted by GitHub as part of the tarball file contents (JuliaLang-Pkg.jl-f9180e4/ vs. JuliaLang-Pkg.jl-d305e82/)

[fredrikekre]

Can we not cache the Pkg-$COMMITSHA folders in SRCCACHE instead of poisoning /stdlib/?

[vtjnash]

If you want an out-of-tree build, you should do an out-of-tree build. Some Packages have a tendency to mutate their own contents (usually just a deps/ folder), so as a whole that probably more nearly makes them build artifacts than source code. But anyways, this variable is just documenting where build artifacts must go (which must not be the same as srccache where source inputs should go); actually deciding whether something is a shared, immutable input or a build artifact is up to the rest of the Makefile.

[fredrikekre]

But why are the checksums also changed? Shouldn't the commit you set here produce the same tarball?

[StefanKarpinski]

Related: #30088, in which the checksums are incorrect.

[StefanKarpinski]

Just to clarify some things that were not clear to me but which I just got clear on talking with @vtjnash:

1. The PKG_SHA1 variable changed from a tree hash to a commit hash with the same tree. This is because you can't git checkout a tree hash but you can do so with a commit and in git checkout mode, git-external uses this name to check things out.

2. The tarball checksums changed because GitHub includes part of PKG_SHA1 in the prefix of paths in the tarball, so paths of things in the tarball changed even though they're otherwise the same.

[KristofferC]

This seems to have broken things. Trying to add Pkg as a dependency gives:

(v1.1) pkg> add Pkg
  Updating registry at `~/.julia/registries/General`
  Updating git-repo `https://github.com/JuliaRegistries/General.git`
ERROR: The following package names could not be resolved:
 * Pkg (44cfe95a-1eb2-52ea-b672-e2afdf69b78f in manifest but not in project)
Please specify by known `name=uuid`.

I also got

  [44cfe95a] - Pkg
→ [44cfe95a] + Pkg-f9180e48b27a843aeee864db814dce57cb296b9b

after doing a Pkg command.

[vtjnash]

The Pkg package shouldn't be able to see this change, but it certainly might expose existing issues (either normalizing with realpath too soon, or looking at the wrong folder altogether).

FWIW, I don't see the issue you describe (although it theoretically would happen if you haven't deleted the usr/ folder in a month, since before the initial PR to move Pkg to be a git-external).

[KristofferC]

The problem with having both Pkg and Pkg-f918.. is that right now there is an assumption that stdlib/StdLib defines a standard library StdLib. This is no longer the case.

[vtjnash]

stdlib is just a random folder, not really associated with anything. The standard library lives inside usr/, as with all build content. Specifically, it's at Sys.STDLIB = ".../usr/share/julia/stdlib/v1.1" (although if you haven't deleted usr/ in some weeks, the build system will be confused and gets them mixed up)