Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Extend Pkg.offline to optionally handle offline registries. Use offline mode during precompile_script #2833

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Extend Pkg.offline to optionally handle offline registries. Use offline mode during precompile_script #2833

wants to merge 1 commit into from

Conversation

IanButterworth
Copy link
Member

@IanButterworth IanButterworth commented Nov 9, 2021 •
edited
Loading

  • Extends Pkg.offline() with a offline_registry kwarg to set whether the registries have offline-accessible paths such that they can still be used to install packages in offline mode
  • Uses this extended offline mode during precompile_script given the mock registry that is set up there uses local source paths. (Using the old Pkg.offline(true) wasn't enough as it disallowed using the registry to Pkg.add a package that wasn't already installed)

Instead of making this a build-specific hack, I thought there may be situations where users have locally linked registries that have local paths, that could be functional during offline mode.

Fixes JuliaLang/julia#43006

Previously lots of server requests were happening. None of these warnings happen now.

Generating REPL precompile statements... 25/36
#### inputting statement: ####
"Pkg.add(\"TestPkg\")"
####
julia> Pkg.add("TestPkg")
   Resolving package versions...
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = Could not resolve host: pkg.julialang.org while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Registry/Registry.jl:76
     Cloning [bddd00b5-e60b-4520-a64a-f93bb8a01829] TestPkg from /var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/TestPkg.jl
   Installed TestPkg ─ v0.1.0
    Updating `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/Project.toml`
  [bddd00b5] + TestPkg v0.1.0
    Updating `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/Manifest.toml`
  [bddd00b5] + TestPkg v0.1.0
  [ade2ca70] + Dates
  [de0858da] + Printf
  [4ec0a83e] + Unicode

Generating REPL precompile statements... 26/36
#### inputting statement: ####
"Pkg.develop(Pkg.PackageSpec(path=\"TestPkg.jl\"))"
####
julia> Pkg.develop(Pkg.PackageSpec(path="TestPkg.jl"))
   Resolving package versions...
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = Could not resolve host: pkg.julialang.org while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Registry/Registry.jl:76
    Updating `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/Project.toml`
  [bddd00b5] ~ TestPkg v0.1.0 ⇒ v0.1.0 `TestPkg.jl`
    Updating `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/Manifest.toml`
  [bddd00b5] ~ TestPkg v0.1.0 ⇒ v0.1.0 `TestPkg.jl`
  [ade2ca70] - Dates
  [de0858da] - Printf
  [4ec0a83e] - Unicode

Generating REPL precompile statements... 27/36
#### inputting statement: ####
"Pkg.add(Pkg.PackageSpec(path=\"TestPkg.jl/\"))"
####
julia> Pkg.add(Pkg.PackageSpec(path="TestPkg.jl/"))
     Cloning git-repo `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/TestPkg.jl`
    Updating git-repo `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/TestPkg.jl`
   Resolving package versions...
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = Could not resolve host: pkg.julialang.org while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Registry/Registry.jl:76
    Updating `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/Project.toml`
  [bddd00b5] ~ TestPkg v0.1.0 `TestPkg.jl` ⇒ v0.1.0 `TestPkg.jl#master`
    Updating `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/Manifest.toml`
  [bddd00b5] ~ TestPkg v0.1.0 `TestPkg.jl` ⇒ v0.1.0 `TestPkg.jl#master`

Generating REPL precompile statements... 28/36
#### inputting statement: ####
"Pkg.REPLMode.try_prompt_pkg_add(Symbol[:notapackage])"
####
julia> Pkg.REPLMode.try_prompt_pkg_add(Symbol[:notapackage])
false

Generating REPL precompile statements... 29/36
#### inputting statement: ####
"Pkg.update()"
####
julia> Pkg.update()
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = Could not resolve host: pkg.julialang.org while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Registry/Registry.jl:76
    Updating registry at `/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/registries/Registry.toml`
    Updating git-repo `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/TestPkg.jl`
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = Could not resolve host: pkg.julialang.org while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Registry/Registry.jl:76
  No Changes to `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/Project.toml`
  No Changes to `/private/var/folders/_6/1yf6sj0950vcg4t91m9ltb5w0000gn/T/jl_FmuO5s/Manifest.toml`
[ Info: We haven't cleaned this depot up for a bit, running Pkg.gc()...
      Active manifest files: 1 found
      Active artifact files: 0 found
      Active scratchspaces: 0 found
     Deleted no artifacts, repos, packages or scratchspaces

Generating REPL precompile statements... 30/36
#### inputting statement: ####
"Pkg.precompile()"
####
julia> Pkg.precompile()
┌ Warning: could not download https://pkg.julialang.org/registries
│   exception = Could not resolve host: pkg.julialang.org while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry ~/Documents/GitHub/julia/usr/share/julia/stdlib/v1.8/Pkg/src/Registry/Registry.jl:76

@KristofferC
Copy link
Member

KristofferC commented Nov 9, 2021 •
edited
Loading

Imo, let's first figure out, why are the requests happening? I don't want to add a API without much thought to fix a small problem in a very specific scenario (the precompile generation script for Julia building).

I guess for Pkg.update it is running the registry updater so we should pass update_registry=false there. For the add I guess it is because we don't set Pkg.UPDATED_REGISTRY_THIS_SESSION. Why is the dev call doing a network request though?

@fredrikekre
Copy link
Member

fredrikekre commented Nov 9, 2021 •
edited
Loading

Why is the dev call doing a network request though?

Probably because auto precompile which now instantiates which resolves which updates? Something like that maybe.

@KristofferC
Copy link
Member

Probably because auto precompile which now instantiates which resolves which updates?

Hm, instantiate should only update a registry if it needs to I think.

@IanButterworth
Copy link
Member Author

The network request is coming from the registry info fetch inside download_source which has caching now, but is never cached when offline so happens every download_source

@KristofferC
Copy link
Member

KristofferC commented Nov 9, 2021 •
edited
Loading

I'm looking at 1.7 (since the issue is on the 1.7) milestone and I don't see that (I guess since #2689 is not on 1.7). For 1.7 #2834 is enough maybe?

@KristofferC KristofferC mentioned this pull request Nov 9, 2021
Merged
@vtjnash
Copy link
Member

vtjnash commented Sep 6, 2023

Bump?

@IanButterworth
Copy link
Member Author

Happy to revive this if people like this direction

@IanButterworth IanButterworth mentioned this pull request Feb 24, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@KristofferC KristofferC Awaiting requested review from KristofferC

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

build requires network access due to Pkg
4 participants
@IanButterworth @KristofferC @fredrikekre @vtjnash