Change ITensorGPU URL

[mtfishman]

No description provided.

[mtfishman]

I don't know how I would go about making the previous versions of the package available using the new URL, since it was a subdirectory package so the development of the package is intertwined with the development of the other packages in the old repository. An extreme solution would be to copy the entire history of the original repository but that seems kind of crazy to me and I'm not even sure that would work.

However, I don't really care very much about making older versions of the package installable, I just plan to register a new patch version in the new repository and make that version available for backwards compatibility, the package is being deprecated anyway.

@giordano sorry to ping you but I've seen that you have been merging these kinds of URL changes.

[giordano]

Can you please verify old versions are still installable? The tree hashes have to be available in the new repository, for the move to work. See for example #35965 (comment)

[mtfishman]

@giordano I'll look into it.

[mtfishman]

@giordano I copied the history of the subdirectory package from the old repository https://github.com/ITensor/ITensors.jl to the new repository https://github.com/ITensor/ITensorGPU.jl.

I've added my fork of the registry with:

using Pkg: Pkg
Pkg.Registry.rm("General")
Pkg.Registry.add(Pkg.Registry.RegistrySpec(url="https://github.com/mtfishman/General"))

following https://pkgdocs.julialang.org/v1/registries/ and then checked out the branch of this PR (itensorgpu_url) with:

cd ~/.julia/registries/General
git checkout itensorgpu_url

I was able to check that the hashes of the old versions exist by running:

$ julia

julia> using TOML: TOML

julia> function check_versions(pkg; registry_name="General")
         versions = TOML.parsefile(joinpath(
           first(DEPOT_PATH),
           "registries",
           registry_name,
           pkg[1:1],
           pkg,
           "Versions.toml",
         ))
         for version in sort(collect(keys(versions)))
           tree_sha = versions[version]["git-tree-sha1"]
           print(version)
           try
             readchomp(`git rev-parse -q --verify "$(tree_sha)^{tree}"`)
             println(" found")
           catch
             println(" missing")
           end
         end
       end
check_versions (generic function with 1 method)

julia> check_versions("ITensorGPU")
0.0.1 found
0.0.2 found
0.0.3 found
0.0.4 found
0.0.5 found
0.0.6 found
0.0.7 found
0.0.8 found
0.1.0 found
0.1.1 found
0.1.2 found
0.1.3 found
0.1.4 found
0.1.5 found
0.1.6 found
0.1.7 found

inside a cloned version of the new repository (https://github.com/ITensor/ITensorGPU.jl).

I've also successfully added arbitrary versions of the ITensorGPU.jl package with:

using Pkg: Pkg
Pkg.add(name="ITensorGPU", version="0.1.1")

The only thing that makes me wonder if I am checking the setup correctly is that when I run Pkg.Registry.status() I get:

julia> Pkg.Registry.status()
Registry Status 
 [23338594] General (https://github.com/JuliaRegistries/General.git)

Shouldn't this show my fork of the registry, i.e. https://github.com/mtfishman/General.git? I've confirmed there is only one registry in ~/.julia/registries and it points to my fork and branch:

$ cd ~/.julia/registries
$ ls
General
$ git remote -v
origin	https://github.com/mtfishman/General (fetch)
origin	https://github.com/mtfishman/General (push)
$ git branch
* itensorgpu_url
  master

For future reference, https://grantwinney.com/how-to-move-a-subdirectory-of-one-repo-into-its-own-repository and https://docs.github.com/en/get-started/using-git/splitting-a-subfolder-out-into-a-new-repository were helpful guides for this kind of transition where you want to transfer a subdirectory package to a new repository where the code that was in the subdirectory is now at the base of the new repository, and additionally you want to preserve the history of only the subdirectory.

[giordano]

Try

]registry rm General

before adding the new one? I really don't know what happens when you add two registries with same name and UUID, that sounds like a not great idea.

[mtfishman]

Before adding my registry fork I removed the old one with:

using Pkg: Pkg
Pkg.Registry.rm("General")
Pkg.Registry.add(Pkg.Registry.RegistrySpec(url="https://github.com/mtfishman/General"))

Is that the correct way to try out my registry fork to test out this PR?

[giordano]

Uhm, try to change name and UUID at https://github.com/mtfishman/General/blob/2c698922808283e361c02753aaa8995718baf1b1/Registry.toml#L1-L2. I wonder if the UUID of General is hardcoded somewhere.

[mtfishman]

I see, thanks for the suggestion.

I tried changing these lines:

General/Registry.toml

Lines 1 to 3 in 207969f

	name = "General"
	uuid = "23338594-aafe-5451-b93e-139f81909106"
	repo = "https://github.com/JuliaRegistries/General.git"

to:

name = "GeneralITensorGPU"
uuid = "4ca3b42f-9706-44c9-ba1f-19f9ce2fad28"
repo = "https://github.com/mtfishman/General.git#itensorgpu_url"

, changing the name and uuid (generated with using UUIDs: UUIDs; UUIDs.uuid4()) to be certain I was using the registry from this PR branch. With that change I can now add this PR's registry with:

$ git clone -b itensorgpu_url https://github.com/mtfishman/General.git GeneralITensorGPU
$ julia

julia> using Pkg: Pkg

julia> Pkg.Registry.rm("General")

julia> Pkg.Registry.add(Pkg.Registry.RegistrySpec(path=joinpath(pwd(), "GeneralITensorGPU"), name="GeneralITensorGPU"))
     Copying registry from `pwd/GeneralITensorGPU`
      Copied registry `GeneralITensorGPU` to `~/.julia/registries/GeneralITensorGPU`

and I can confirm that I am using the registry from this PR:

julia> using Pkg: Pkg

julia> Pkg.Registry.status()
Registry Status 
 [4ca3b42f] GeneralITensorGPU (https://github.com/mtfishman/General.git#itensorgpu_url)

After doing that, I checked that:

julia> check_versions("ITensorGPU"; registry_name="GeneralITensorGPU")
0.0.1 found
0.0.2 found
0.0.3 found
0.0.4 found
0.0.5 found
0.0.6 found
0.0.7 found
0.0.8 found
0.1.0 found
0.1.1 found
0.1.2 found
0.1.3 found
0.1.4 found
0.1.5 found
0.1.6 found
0.1.7 found

when called from the base of the ITensorGPU.jl repository and that I can add arbitrary versions of ITensorGPU.jl with using Pkg: Pkg; Pkg.add(name="ITensorGPU", version="0.1.1").

As an extra check, I confirmed that if I intentionally set an incorrect URL, for example:
https://github.com/mtfishman/General/blob/eb227b174fd1164ee71285fbbda821fa34748d6c/I/ITensorGPU/Package.toml#L3
then I can no longer install ITensorGPU.jl.

[mtfishman]

@giordano I think this is ready to merge, I believe I've tested it as much as I can.

As a more manual sanity check, I've looked up the tree hash for a few commits corresponding to the same registered ITensorGPU.jl version in each repository.

For example, the tree hash for ITensorGPU.jl v0.1.6 is here:

General/I/ITensorGPU/Versions.toml

Line 44 in 00ae226

git-tree-sha1 = "57114051942fd0e74bf41cf66982d879af66b558"

The corresponding commit in the original repository is here:
ITensor/ITensors.jl@73bc07e
Executing the following at the base of the cloned https://github.com/ITensor/ITensors.jl repository gives:

$ git cat-file -p 73bc07eded9bd5f88da1a1562986ac5fb6e42a10
tree 08ae5b00cb5343d0b56fce41a70d2f7146b7fc3b
# ...

$ git cat-file -p 08ae5b00cb5343d0b56fce41a70d2f7146b7fc3b
100644 blob 08f664cdb99980d02a9840b16f74f56f2731acdc	.JuliaFormatter.toml
040000 tree ee99a5fd9b2c9acd1ddb3a07fee9d11791eaa573	.github
100644 blob ac4095b6a18c42f688d654aeb4aff43ea87d4a17	.gitignore
100644 blob 411faaf5f4fa714b3e7777d6043dfb6e118dbc9e	CITATION.cff
100644 blob c79a13ca0b0314a5c0d181eae9398c6b5ae49d3e	CODE_OF_CONDUCT.md
100644 blob e29ef78712568c12a2ce5d091825aef644012bf8	Checklists.txt
040000 tree f90dc6bbb991f0bac3d6065beda1cf6750e6ac96	ITensorGLMakie
040000 tree 57114051942fd0e74bf41cf66982d879af66b558	ITensorGPU
040000 tree 9666aa81a71296a0dfc6f86a0893f1e9721051eb	ITensorGaussianMPS
040000 tree dbfc4f32f5cec3505879fb3649781beb67384975	ITensorMakie
040000 tree 19e304eb8d77ed86cd0e73707f650f82fb0bff28	ITensorUnicodePlots
040000 tree edc9be09004c1d8540e42bf4dc7ca00e5230801a	ITensorVisualizationBase
100644 blob 555297e50a6fd2b2c9058abf6b02eb8ae62efc2f	LICENSE
040000 tree 2a5191a7aa04f6c10826231cccc2835fa5dbacb0	NDTensors
100644 blob 8c249d6abac8fe6b28b7c705e38859d6a510942c	NEWS.md
100644 blob a221bd65dd49b8777cde114b603c38b62dbab9da	Project.toml
120000 blob 8785db193efea5e355d1bb2800d28ca683261dd3	README.md
040000 tree 8092d48e92b6e2e3a7c371be17ad5d659a8b2d5c	benchmark
040000 tree adea9810bf4ccf5cf02794d705074dfceb46d967	docs
040000 tree f95c6ebf10d075eee8a9acfd7b29dd9e5051e9ab	examples
040000 tree e8b4df3504595bea30f122bb592df1d7598abd00	jenkins
040000 tree 5eb867b484d6b205c4437ee4fb5897ac76a73b9a	precompile
040000 tree e75e8f6f0386d00d7d8cf3d0f8ddb304f4e6a640	src
040000 tree 8f1afc444ac2fd57902ceaa40fe31d272592c9ac	test

so the tree hash for the ITensorGPU subdirectory for ITensorGPU.jl v0.1.6 is 57114051942fd0e74bf41cf66982d879af66b558, which matches the tree hash in the registry for ITensorGPU.jl v0.1.6.

The corresponding commit in the new repository is here:
ITensor/ITensorGPU.jl@e08600e
Executing the following at the base of the cloned https://github.com/ITensor/ITensorGPU.jl repository gives:

$ git cat-file -p e08600e3599729cc9850db3df8095abeeac43b22
tree 57114051942fd0e74bf41cf66982d879af66b558

which also matches the tree hash in the registry for ITensorGPU.jl v0.1.6.

[mtfishman]

CONTRIBUTING.md has an appendix on checking the registered versions of a repository (introduced in #75328) which provides a simple script that doesn't involve loading a registry branch as a package registry as I struggled but managed to do above. Loading that script and running the function check_package_versions(pkg_name, repo_url) gives:

julia> check_package_versions("ITensorGPU", "https://github.com/ITensor/ITensorGPU.jl.git")
Cloning into '/var/folders/qz/q22pzwm144z9fq57mpf1hfp40000gq/T/jl_Bv02LL'...
remote: Enumerating objects: 325, done.
remote: Counting objects: 100% (325/325), done.
remote: Compressing objects: 100% (130/130), done.
remote: Total 325 (delta 191), reused 325 (delta 191), pack-reused 0
Receiving objects: 100% (325/325), 83.25 KiB | 7.57 MiB/s, done.
Resolving deltas: 100% (191/191), done.
ITensorGPU: v0.0.1 found
ITensorGPU: v0.0.2 found
ITensorGPU: v0.0.3 found
ITensorGPU: v0.0.4 found
ITensorGPU: v0.0.5 found
ITensorGPU: v0.0.6 found
ITensorGPU: v0.0.7 found
ITensorGPU: v0.0.8 found
ITensorGPU: v0.1.0 found
ITensorGPU: v0.1.1 found
ITensorGPU: v0.1.2 found
ITensorGPU: v0.1.3 found
ITensorGPU: v0.1.4 found
ITensorGPU: v0.1.5 found
ITensorGPU: v0.1.6 found
ITensorGPU: v0.1.7 found
16-element Vector{@NamedTuple{pkg_name::String, version::VersionNumber, found::Bool}}:
 (pkg_name = "ITensorGPU", version = v"0.0.1", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.0.2", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.0.3", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.0.4", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.0.5", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.0.6", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.0.7", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.0.8", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.1.0", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.1.1", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.1.2", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.1.3", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.1.4", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.1.5", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.1.6", found = 1)
 (pkg_name = "ITensorGPU", version = v"0.1.7", found = 1)

so that is another confirmation that the new repository https://github.com/ITensor/ITensorGPU.jl.git is set up properly.

Sorry for the long posts, part of this is meant to serve as a reference for future Julia developers (include my future self) to navigate doing these kind of advanced package registration processes. I remember coming across that appendix at some point but didn't fully appreciate what its purpose was and then couldn't find it again until now. I think that cleanly solves the issue of checking that package URL changes are implemented properly.

The hardest part of all this is doing git magic to change the URL of a Julia package while preserving the tree hashes¹, especially if it involves subdirectory packages. I hadn't even conceived of the fact that it was possible to preserve the git history of just a subdirectory of a repository (I assumed that would be very difficult since my only mental model for git is based on commit hashes), and I struggled to come up with the right terminology to search for or understand the guides and documentation about how to do that, a common git issue in my experience. As a summary, the key for me was to follow guides like https://grantwinney.com/how-to-move-a-subdirectory-of-one-repo-into-its-own-repository and https://docs.github.com/en/get-started/using-git/splitting-a-subfolder-out-into-a-new-repository. This comment: #35965 (comment) mentions providing a tutorial about that somewhere, which would be helpful (even just pointing to the Github documentation would be helpful to get people started).

Footnotes

1. Or even learning what tree hashes were and that they, not commit hashes, are used to uniquely identify package versions by the Julia registry, quite clever for handling these kinds of use cases! ↩

[giordano]

Thank you so much for all the tests! Please, if you believe the documentation in CONTRIBUTING.md can be improved to make life easier for others in the future, please go ahead and make a PR!

[mtfishman]

Thanks for the review! I'll make a PR adding a few clarifications. I'm going through this process transferring a few other subdirectory packages to separate repositories so I'll try to hone my workflow so I can make more succinct instructions.