Unable to install unregistered package in Julia >=1.7 (when providing a Context) #3251

fcdimitr · 2022-11-04T23:12:28Z

Installing an unregistered package currently fails. I am attaching a MWE:

using Pkg
Pkg.add(Pkg.Types.Context(), [PackageSpec(; name = "Box2D_jll", uuid = "6756b63d-2e54-5dc6-816b-1fc83e1ae834", url = "https://github.com/cirocavani/Box2D_jll.jl")])

which results in the following error:

    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General`
   Resolving package versions...
ERROR: expected package `Box2D_jll [6756b63d]` to be registered
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
 [2] check_registered
   @ /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1190 [inlined]
 [3] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
   @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1252
 [4] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber)
   @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1225
 [5] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
   @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1260
 [6] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
   @ Pkg.Operations /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/Operations.jl:1276
 [7] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/API.jl:275
 [8] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
   @ Pkg.API /Applications/Julia-1.8.app/Contents/Resources/julia/share/julia/stdlib/v1.8/Pkg/src/API.jl:226
 [9] top-level scope
   @ REPL[4]:1

The output of versioninfo():

Julia Version 1.8.2
Commit 36034abf260 (2022-09-29 15:21 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin21.4.0)
  CPU: 4 × Intel(R) Core(TM) i7-4558U CPU @ 2.80GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-13.0.1 (ORCJIT, haswell)
  Threads: 1 on 4 virtual cores
Environment:
  JULIA_PKG_DEVDIR = /Users/fcdimitr/Workspace/julia-dev

Julia is installed as a cask via homebrew.

The issue also appears on a Linux machine. It has also been discussed at JuliaPackaging/BinaryBuilder.jl#1173

The text was updated successfully, but these errors were encountered:

KristofferC · 2022-11-04T23:15:27Z

Remove the context argument. If you have to use it (knowing it is an implementation detail) then you need to read the source to know how to use it properly.

fcdimitr · 2022-11-05T00:02:55Z

Thank you, @KristofferC! I am trying to understand the issue with BinaryBuilder; I am not using this directly in my module. I will continue the discussion there and update this issue appropriately.

fcdimitr · 2022-11-05T01:26:50Z

It seems that using the context argument is a requirement of the BinaryBuilder.jl package.

I do not know why and how it is used. If I understand more about the internals of BinaryBuilder.jl, I will follow up here.

Thank you!

sloede · 2023-03-20T13:51:46Z

This issue severely degrades the ability to use BB.jl and Julia for rapid prototyping with existing C/C++ libraries in a reproducible manner. Is there any way this will be fixed in the foreseeable future? I know everyone is busy, thus I am not complaining, just kindly asking to know what the status is 🙂

IanButterworth · 2023-03-20T14:03:31Z

This issue needs more information on why the given internal (evidently incorrect) format is being used, and why this cannot simply be used.

julia> Pkg.add(url = "https://github.com/cirocavani/Box2D_jll.jl")
     Cloning git-repo `https://github.com/cirocavani/Box2D_jll.jl`
    Updating git-repo `https://github.com/cirocavani/Box2D_jll.jl`
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
  Downloaded artifact: Box2D
  Downloaded artifact: Box2D
    Updating `~/.julia/environments/v1.8/Project.toml`
  [6756b63d] + Box2D_jll v2.4.1+0 `https://github.com/cirocavani/Box2D_jll.jl#main`
    Updating `~/.julia/environments/v1.8/Manifest.toml`
  [6756b63d] + Box2D_jll v2.4.1+0 `https://github.com/cirocavani/Box2D_jll.jl#main`
Precompiling project...
  4 dependencies successfully precompiled in 5 seconds. 51 already precompiled.

on julia 1.8.5

sloede · 2023-03-20T14:14:21Z

This issue needs more information on why the given internal (evidently incorrect) format is being used, and why this cannot simply be used.

According to @giordano (not pointing fingers, just citing the source), the context argument seems to be required by BinaryBuilder.jl (JuliaPackaging/BinaryBuilder.jl#1173 (comment)). Unfortunately, I am completely out of my depth to say whether this is correct or what do do about it.

I am just observing that it used to work for BB with unregistered packages, now it doesn't, and that it seems to be related to a change in Pkg.

IanButterworth · 2023-03-20T14:20:06Z

For the record dropping the name and uuid causes another issue, that the error message seems to imply should be supported

julia> Pkg.add(Pkg.Types.Context(), [Pkg.Types.PackageSpec(; url = "https://github.com/cirocavani/Box2D_jll.jl")])
ERROR: name, UUID, URL, or filesystem path specification required when calling `add`
Stacktrace:
 [1] pkgerror(msg::String)
   @ Pkg.Types ~/.julia/juliaup/julia-1.8.5+0.x64.apple.darwin14/share/julia/stdlib/v1.8/Pkg/src/Types.jl:67
 [2] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ Pkg.API ~/.julia/juliaup/julia-1.8.5+0.x64.apple.darwin14/share/julia/stdlib/v1.8/Pkg/src/API.jl:237
 [3] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec})
   @ Pkg.API ~/.julia/juliaup/julia-1.8.5+0.x64.apple.darwin14/share/julia/stdlib/v1.8/Pkg/src/API.jl:226
 [4] top-level scope
   @ REPL[8]:1

IanButterworth · 2023-03-20T14:24:47Z

You need to call handle_package_input! on the list of packages

julia> pkgs = [Pkg.Types.PackageSpec(; name = "Box2D_jll", uuid = "6756b63d-2e54-5dc6-816b-1fc83e1ae834", url = "https://github.com/cirocavani/Box2D_jll.jl")]
1-element Vector{Pkg.Types.PackageSpec}:
 PackageSpec(
  name = Box2D_jll
  uuid = 6756b63d-2e54-5dc6-816b-1fc83e1ae834
  url = https://github.com/cirocavani/Box2D_jll.jl
  version = *
)

julia> foreach(Pkg.API.handle_package_input!, pkgs)

julia> pkgs
1-element Vector{Pkg.Types.PackageSpec}:
 PackageSpec(
  name = Box2D_jll
  uuid = 6756b63d-2e54-5dc6-816b-1fc83e1ae834
  url = https://github.com/cirocavani/Box2D_jll.jl
  version = *
  repo/source = "https://github.com/cirocavani/Box2D_jll.jl"
)

julia> Pkg.add(Pkg.Types.Context(), pkgs)
    Updating git-repo `https://github.com/cirocavani/Box2D_jll.jl`
   Resolving package versions...
  No Changes to `~/.julia/environments/v1.8/Project.toml`
  No Changes to `~/.julia/environments/v1.8/Manifest.toml`

giordano · 2023-03-20T14:27:55Z

You need to call handle_package_input! on the list of packages

I was waiting for someone to suggest that. No, it doesn't work: JuliaPackaging/BinaryBuilderBase.jl#212

IanButterworth · 2023-03-20T14:30:22Z

If that's the case then it would be good to have a MWE that it doesn't work for.

I think a lot of the BB issues are because providing a Context() (which is intentionally an internal mode) skips over the conditioning that happens for the majority of the API functions here, the key being handle_package_input!

Pkg.jl/src/API.jl

Lines 141 to 180 in 1c6ba55

    
           for f in (:develop, :add, :rm, :up, :pin, :free, :test, :build, :status, :why, :precompile) 
        
               @eval begin 
        
                   $f(pkg::Union{AbstractString, PackageSpec}; kwargs...) = $f([pkg]; kwargs...) 
        
                   $f(pkgs::Vector{<:AbstractString}; kwargs...)          = $f([PackageSpec(pkg) for pkg in pkgs]; kwargs...) 
        
                   function $f(pkgs::Vector{PackageSpec}; io::IO=$(f === :status ? :stdout_f : :stderr_f)(), kwargs...) 
        
                       $(f != :precompile) && Registry.download_default_registries(io) 
        
                       ctx = Context() 
        
                       # Save initial environment for undo/redo functionality 
        
                       if $(f != :precompile) && !saved_initial_snapshot[] 
        
                           add_snapshot_to_undo(ctx.env) 
        
                           saved_initial_snapshot[] = true 
        
                       end 
        
                       kwargs = merge((;kwargs...), (:io => io,)) 
        
                       pkgs = deepcopy(pkgs) # don't mutate input 
        
                       foreach(handle_package_input!, pkgs) 
        
                       ret = $f(ctx, pkgs; kwargs...) 
        
                       $(f in (:add, :up, :pin, :free, :build)) && Pkg._auto_precompile(ctx) 
        
                       $(f in (:up, :pin, :free, :rm)) && Pkg._auto_gc(ctx) 
        
                       return ret 
        
                   end 
        
                   $f(ctx::Context; kwargs...) = $f(ctx, PackageSpec[]; kwargs...) 
        
                   function $f(; name::Union{Nothing,AbstractString}=nothing, uuid::Union{Nothing,String,UUID}=nothing, 
        
                                 version::Union{VersionNumber, String, VersionSpec, Nothing}=nothing, 
        
                                 url=nothing, rev=nothing, path=nothing, mode=PKGMODE_PROJECT, subdir=nothing, kwargs...) 
        
                       pkg = PackageSpec(; name, uuid, version, url, rev, path, subdir) 
        
                       if $f === status || $f === rm || $f === up 
        
                           kwargs = merge((;kwargs...), (:mode => mode,)) 
        
                       end 
        
                       # Handle $f() case 
        
                       if all(isnothing, [name,uuid,version,url,rev,path,subdir]) 
        
                           $f(PackageSpec[]; kwargs...) 
        
                       else 
        
                           $f(pkg; kwargs...) 
        
                       end 
        
                   end 
        
                   function $f(pkgs::Vector{<:NamedTuple}; kwargs...) 
        
                       $f([PackageSpec(;pkg...) for pkg in pkgs]; kwargs...) 
        
                   end 
        
               end 
        
           end

sloede · 2023-03-20T14:49:07Z

You need to call handle_package_input! on the list of packages

Based on your suggestion, I found a workaround for BB until the underlying issue is fixed: JuliaPackaging/BinaryBuilder.jl#1173 (comment)

youwuyou · 2023-10-07T06:25:20Z

This issue severely degrades the ability to use BB.jl and Julia for rapid prototyping with existing C/C++ libraries in a reproducible manner. Is there any way this will be fixed in the foreseeable future? I know everyone is busy, thus I am not complaining, just kindly asking to know what the status is 🙂

Same opinion too :D would be amazing if it could be fixed in the near future

fcdimitr mentioned this issue Nov 5, 2022

Non-registered dependency not working with Julia 1.7 JuliaPackaging/BinaryBuilder.jl#1173

Open

giordano mentioned this issue Nov 11, 2022

Support for Apple M1 jump-dev/Clp.jl#131

Open

IanButterworth changed the title ~~Unable to install unregistered package in Julia >=1.7~~ Unable to install unregistered package in Julia >=1.7 (when providing a Context) Mar 20, 2023

giordano added the binarybuilder Related to the use of Pkg in BinaryBuilder label Mar 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to install unregistered package in Julia >=1.7 (when providing a Context) #3251

Unable to install unregistered package in Julia >=1.7 (when providing a Context) #3251

fcdimitr commented Nov 4, 2022

KristofferC commented Nov 4, 2022

fcdimitr commented Nov 5, 2022

fcdimitr commented Nov 5, 2022

sloede commented Mar 20, 2023

IanButterworth commented Mar 20, 2023 •

edited

Loading

sloede commented Mar 20, 2023

IanButterworth commented Mar 20, 2023

IanButterworth commented Mar 20, 2023 •

edited

Loading

giordano commented Mar 20, 2023

IanButterworth commented Mar 20, 2023 •

edited

Loading

sloede commented Mar 20, 2023

youwuyou commented Oct 7, 2023

Unable to install unregistered package in Julia >=1.7 (when providing a Context) #3251

Unable to install unregistered package in Julia >=1.7 (when providing a Context) #3251

Comments

fcdimitr commented Nov 4, 2022

KristofferC commented Nov 4, 2022

fcdimitr commented Nov 5, 2022

fcdimitr commented Nov 5, 2022

sloede commented Mar 20, 2023

IanButterworth commented Mar 20, 2023 • edited Loading

sloede commented Mar 20, 2023

IanButterworth commented Mar 20, 2023

IanButterworth commented Mar 20, 2023 • edited Loading

giordano commented Mar 20, 2023

IanButterworth commented Mar 20, 2023 • edited Loading

sloede commented Mar 20, 2023

youwuyou commented Oct 7, 2023

IanButterworth commented Mar 20, 2023 •

edited

Loading

IanButterworth commented Mar 20, 2023 •

edited

Loading

IanButterworth commented Mar 20, 2023 •

edited

Loading