Feature request: scan subdirs for updated versions/projects

[timholy]

Summary

For repositories that have more than one package on them, and some of the packages depend on others, CI runs into trouble because julia --project picks the "umbrella" Project.toml files which sets a new version just for that one package. If the umbrella package depends on new versions of the subdirectory packages, the resolver doesn't know that it can get them from the current PR.

Current workaround

If you're running CI with GitHub Actions, add something like

defaults:
  run:
    shell: bash
...

    steps:
      ...
      - run: julia --project -e 'using Pkg; Pkg.develop([PackageSpec(path="SubDirA"),
                                                         PackageSpec(path="SubDirB"), ...])'

Travis users should add something similar.

Reproducer

Following on the lengthy discussion in #1251, I thought since all the people who actually know Pkg internals were confused about what I was asking, I would create something that is close to an actual test case (and therefore may assist in developing the requested feature). The executive summary here is that Pkg does not yet support updating its awareness of sub-package versions based on a new pull request.

Setup: create a blank directory to do this test in, I chose /tmp/pkgs. Save the script below as /tmp/pkgs/generate_test.jl, make sure you've cd /tmp/pkgs, launch Julia as JULIA_DEPOT_PATH=/tmp/pkgs julia --startup-file=no (I was using Julia 1.6 just to get the latest) and then include the script. Here's the script:

function writelines(path, lines)
    open(path, "w") do io
        for line in lines
            println(io, line)
        end
    end
end

# Create a package BigPkg with a subdirectory SubPkg
# Make BigPkg depend on SubPkg
# Register it
using Pkg
Pkg.add("LocalRegistry")
using LocalRegistry
regpath = joinpath(@__DIR__, "registries", "MyRegistry")
create_registry("MyRegistry", regpath)
# Pkg.Registry.add(regpath)
devpath = joinpath(@__DIR__, "dev")
mkpath(devpath)
bigpkgpath = joinpath(devpath, "BigPkg")
subpkgpath = joinpath(bigpkgpath, "SubPkg")
Pkg.generate(bigpkgpath)
Pkg.generate(subpkgpath)
Pkg.develop(path=bigpkgpath)
Pkg.develop(path=subpkgpath)
cd(bigpkgpath) do
    run(`git init`)
    run(`git add Project.toml src/BigPkg.jl SubPkg/Project.toml SubPkg/src/SubPkg.jl`)
    run(`git commit -m "Skeleton"`)

    Pkg.activate(".")
    Pkg.add(path=".", subdir="SubPkg")
    lines = readlines("Project.toml")
    append!(lines, ["", "[compat]", "SubPkg = \"0.1\""])
    writelines("Project.toml", lines)
    lines = readlines("src/BigPkg.jl")
    insert!(lines, 2, "using SubPkg")
    writelines("src/BigPkg.jl", lines)
    Pkg.activate()
    Pkg.resolve()
    run(`git add Project.toml src/BigPkg.jl`)
    run(`git commit -m "Add SubPkg dependency"`)
end
using BigPkg
using SubPkg
register(SubPkg, "MyRegistry"; repo=bigpkgpath)
register(BigPkg, "MyRegistry"; repo=bigpkgpath)

# OK, now both packages are registered at v0.1.
# Let's add a new feature to SubPkg and bump its version
# to 0.2. Simultaneously, let's require the new feature
# in BigPkg and bump its version to 0.2.
cd(subpkgpath) do
    lines = readlines("src/SubPkg.jl")
    insert!(lines, 2, "export f")
    insert!(lines, 3, "f() = 1")
    writelines("src/SubPkg.jl", lines)
    lines = readlines("Project.toml")
    idx = findfirst(str->startswith(str, "version"), lines)
    lines[idx] = "version = \"0.2.0\""
    writelines("Project.toml", lines)
end
cd(bigpkgpath) do
    lines = readlines("src/BigPkg.jl")
    insert!(lines, 3, "g() = f()")
    writelines("src/BigPkg.jl", lines)
    lines = readlines("Project.toml")
    idx = findfirst(str->startswith(str, "version"), lines)
    lines[idx] = "version = \"0.2.0\""
    lines[end] = "SubPkg = \"0.2\""
    writelines("Project.toml", lines)
    mkdir("test")
    open("test/runtests.jl", "w") do io
        println(io, """
        using BigPkg, Test
        @test BigPkg.g() == 1
        """)
    end

    run(`git add Project.toml src/BigPkg.jl SubPkg/Project.toml SubPkg/src/SubPkg.jl test/runtests.jl`)
    run(`git commit -m "Version 0.2"`)
end

# Remove packages in preparation for simulating what happens when we
# submit a PR and CI runs on Travis
Pkg.rm("SubPkg")
Pkg.rm("BigPkg")

Then I navigate to dev and do this:

tim@diva:/tmp/pkgs/dev$ JULIA_DEPOT_PATH=/tmp/pkgs JL_PKG=BigPkg julia-master --startup-file=no -e 'using Pkg; Pkg.build(verbose=true)'

For some reason, executing the tests as in the default travis script doesn't seem to work for me, but I can do this:

tim@diva:/tmp/pkgs/dev/BigPkg$ JULIA_DEPOT_PATH=/tmp/pkgs julia-master --startup-file=no --project -e 'using Pkg; Pkg.test("BigPkg", coverage=false)'
    Testing BigPkg
┌ Error: Pkg.Resolve.ResolverError("empty intersection between [email protected] and project compatibility 0.2", nothing)
└ @ Pkg.Operations ~/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1410
ERROR: empty intersection between SubPkg@0.1.0 and project compatibility 0.2
Stacktrace:
 [1] resolve_versions!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:370
 [2] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Pkg.Types.UpgradeLevel) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1209
 [3] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:245
 [4] up(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:level, :mode, :update_registry, :io),Tuple{Pkg.Types.UpgradeLevel,Pkg.Types.PackageMode,Bool,Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:68
 [5] resolve(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:251
 [6] (::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec})() at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1412
 [7] with_temp_env(::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec}, ::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1331
 [8] (::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String})(::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1402
 [9] mktempdir(::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String}, ::String; prefix::String) at ./file.jl:682
 [10] mktempdir(::Function, ::String) at ./file.jl:680 (repeats 2 times)
 [11] sandbox(::Function, ::Pkg.Types.Context, ::Pkg.Types.PackageSpec, ::String, ::String, ::Pkg.Types.Project) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1369
 [12] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1533
 [13] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:327
 [14] #test#61 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:67 [inlined]
 [15] #test#60 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:66 [inlined]
 [16] test(::String; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:coverage,),Tuple{Bool}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:65
 [17] top-level scope at none:1
caused by [exception 1]
empty intersection between SubPkg@0.1.0 and project compatibility 0.2
Stacktrace:
 [1] resolve_versions!(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:370
 [2] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Pkg.Types.UpgradeLevel) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1209
 [3] up(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, update_registry::Bool, kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:245
 [4] up(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Any,NTuple{4,Symbol},NamedTuple{(:level, :mode, :update_registry, :io),Tuple{Pkg.Types.UpgradeLevel,Pkg.Types.PackageMode,Bool,Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:68
 [5] resolve(::Pkg.Types.Context; kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:251
 [6] resolve(; kwargs::Base.Iterators.Pairs{Symbol,Base.DevNull,Tuple{Symbol},NamedTuple{(:io,),Tuple{Base.DevNull}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:249
 [7] (::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec})() at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1407
 [8] with_temp_env(::Pkg.Operations.var"#98#102"{String,Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.PackageSpec}, ::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1331
 [9] (::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String})(::String) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1402
 [10] mktempdir(::Pkg.Operations.var"#97#101"{Pkg.Operations.var"#108#110"{Bool,Cmd,Cmd,Nothing,Pkg.Types.Context,Array{Tuple{String,Base.Process},1},String,Pkg.Types.PackageSpec},Pkg.Types.Context,Pkg.Types.PackageSpec,String,Pkg.Types.Project,String}, ::String; prefix::String) at ./file.jl:682
 [11] mktempdir(::Function, ::String) at ./file.jl:680 (repeats 2 times)
 [12] sandbox(::Function, ::Pkg.Types.Context, ::Pkg.Types.PackageSpec, ::String, ::String, ::Pkg.Types.Project) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1369
 [13] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1533
 [14] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:327
 [15] #test#61 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:67 [inlined]
 [16] #test#60 at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:66 [inlined]
 [17] test(::String; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:coverage,),Tuple{Bool}}}) at /home/tim/src/julia-master/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:65
 [18] top-level scope at none:1

So, the obvious problem is that SubPkg is a dependency of BigPkg, and this new version of BigPkg requires [email protected], but [email protected] has not been registered yet.

In #1251, @fredrikekre correctly points out that this is no different from how it works normally. But when you have long dependency chains, how it works normally is really painful. When you have a single git repo, you would like to be able to leverage that organization to develop all the packages in concert. To the developer, the most sensible unit is the repo, not the package.

What do we need? I think the only thing missing is some command that will update the session's notion of what package versions are available by scanning the Project.toml files of all packages listed in the [deps] section of BigPkg. If it does that, and then discovers that in the checked-out copy, lo and behold there is a [email protected], then it can proceed and try everything together.

[timholy]

I'm trying to work around this issue by modifying my ci.yml file (GitHub Actions) so that the relevant section reads

    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
          arch: ${{ matrix.arch }}
      - run: julia -e 'using Pkg; pkg"dev SubPkg"'
      - uses: julia-actions/julia-buildpkg@latest
      - uses: julia-actions/julia-runtest@latest

Sadly, it's not working. Any thoughts? I could go back to adding a build/deps.jl step but that will affect users if I don't forget to strip it out before I ship.

Here's the stacktrace:

ERROR: MethodError: no method matching getindex(::Nothing, ::String)
Stacktrace:
 [1] #build_versions#47(::Bool, ::Function, ::Pkg.Types.Context, ::Array{Base.UUID,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1044
 [2] build_versions at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1033 [inlined]
 [3] #add_or_develop#62(::Array{Base.UUID,1}, ::Symbol, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/Operations.jl:1204
 [4] #add_or_develop at ./none:0 [inlined]
 [5] #add_or_develop#15(::Symbol, ::Bool, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/API.jl:69
 [6] (::getfield(Pkg.API, Symbol("#kw##add_or_develop")))(::NamedTuple{(:mode,),Tuple{Symbol}}, ::typeof(Pkg.API.add_or_develop), ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at ./none:0
 [7] do_develop!(::Dict{Symbol,Any}, ::Array{Pkg.Types.PackageSpec,1}, ::Dict{Symbol,Any}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:714
 [8] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Function, ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:697
 [9] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:696
 [10] do_cmd!(::Pkg.REPLMode.PkgCommand, ::Pkg.REPLMode.MiniREPL) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:603
 [11] #do_cmd#33(::Bool, ::Function, ::Pkg.REPLMode.MiniREPL, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.0/Pkg/src/REPLMode.jl:577
 [12] (::getfield(Pkg.REPLMode, Symbol("#kw##do_cmd")))(::NamedTuple{(:do_rethrow,),Tuple{Bool}}, ::typeof(Pkg.REPLMode.do_cmd), ::Pkg.REPLMode.MiniREPL, ::String) at ./none:0
 [13] top-level scope at none:0

[KristofferC]

The code to repro this seems to not work 100%? It gave me a Project.toml that looked like

name = "BigPkg"
uuid = "e4dbb02c-8206-4006-8b08-6815cbd025b8"
SubPkg = "0.2"

I guess it should be something like

name = "BigPkg"
uuid = "e4dbb02c-8206-4006-8b08-6815cbd025b8"

[deps]
SubPkg = "acd...."

[compat]
SubPkg = "0.2"

Couldn't you check in a manifest in BigPkg where all subdir dependencies are devved? That way you always run CI on the "state of the repo". You could run Pkg.update before testing if you want to use the last version of the other dependencies.

[timholy]

Yes, that looks borked. Strange, it worked perfectly for me. I just tested again, and again it was fine.

The Manifest solution seems to work, many thanks for suggesting what should have been obvious! I have actively avoided them for various reasons (worries about testing stale rather than latest versions of pkgs, concerns about how well they generalize across Julia versions, etc), but this does sound like an excellent workaround. I think we still might consider not making an Manifest.toml obligate, but for the time being this is a simple solution.

[timholy]

Is it possible to add multiple Manifest files for different Julia versions? We're getting CI failures due to what appear to be Julia-version [compat] requirements of some of our dependencies. Yes, we could manually pick an earlier version, but it seems likely that there will be some packages for which there is no single version that works across all Julia versions.

[StefanKarpinski]

There's no way to do that currently, and I'm not sure we want to go down that path. I could turn out to be wrong but it feels like one of those annoyances that may sort itself out over time if we suffer through the discomfort for a while and figure out other ways to deal with it. If you're running code against multiple different Julia versions, isn't it better to not check the manifest in and let the resolver pick versions that are compatible with your Julia version?

[timholy]

Currently that's the only way I know of, other than adding a deps/build.jl script that devs subdirectory-packages (which is a worse solution), of solvingcircumventing this issue. But I'd rather not check in the Manifest.toml at all, frankly.

[KristofferC]

You could dev the sub-packages before running the tests in CI perhaps?

[aminya]

You could dev the sub-packages before running the tests in CI perhaps?

I am confused a bit. This is similar to timholy/SnoopCompile.jl@bb56316 and my whole issue on it #1828.

If running this script is a bad thing we should not do it at all (including in CI). If it is a good thing why we don't do it by default.

[fredrikekre]

You could dev the sub-packages before running the tests in CI perhaps?

I am confused a bit. This is similar to timholy/SnoopCompile.jl@bb56316 and my whole issue on it #1828

No, the dev suggestion is part of installing the package, not some runtime adding of packages.

[aminya]

Both suggestions that I made were are also meant for the installation phase of a package.

[fredrikekre]

No they would be executed after installation. The build script runs after all packages have been resolved. You should not run Pkg operations in the build script and timholy/SnoopCompile.jl@bb56316 is invalid.

[timholy]

Worth mentioning that trying to do this from the ci.yml file is not yet working. Any ideas? timholy/SnoopCompile.jl#107

[aminya]

Worth mentioning that trying to do this from the ci.yml file is not yet working. Any ideas? timholy/SnoopCompile.jl#107

That is probably the Powershell issue I gave a solution for here.

[aminya]

No they would be executed after installation. The build script runs after all packages have been resolved. You should not run Pkg operations in the build script and timholy/SnoopCompile.jl@bb56316 is valid.

I have no objection to running it before doing any Pkg operation. Project.jl could run before installation. Here, I abused deps/build.jl as the only possible way of installing something, as I mentioned in #1828 (comment)

You could dev the sub-packages before running the tests in CI perhaps?

Now, I need to dev these packages everywhere that I want to test a developing SnoopCompile branch. Pkg.add simply does not work. See
timholy/SnoopCompile.jl#107 (comment)

[fredrikekre]

Now, I need to dev these packages everywhere that I want to test a developing SnoopCompile branch.

No, this is only because those packages are not registered yet. When they are you only have to dev SnoopCompile to develop SnoopCompile, just like you only need to dev DataFrames to develop DataFrames.

[KristofferC]

I must say that even with this dedicated issue I have a bit of a hard time wrapping my head about what needs to be done and what

I think the only thing missing is some command that will update the session's notion of what package versions are available by scanning the Project.toml files of all packages listed in the [deps] section of BigPkg. If it does that, and then discovers that in the checked-out copy, lo and behold there is a [email protected], then it can proceed and try everything together.

actually means. This is literally what happens when a package is developed and you run resolve so I don't understand why that is not the solution. If you want to track packages by local files you use dev, if you want to track packages by registered versions you use add.

[timholy]

(@v1.6) pkg> st SnoopCompile SnoopCompileCore SnoopCompileAnalysis SnoopCompileBot
Status `~/.julia/environments/v1.6/Project.toml`
  [aa65fe97] SnoopCompile v1.6.0
  [9ea4277c] SnoopCompileAnalysis v1.6.0
  [1d5e0e55] SnoopCompileBot v1.6.0
  [e2b509da] SnoopCompileCore v1.6.0

(@v1.6) pkg> dev SnoopCompile
Path `/home/tim/.julia/dev/SnoopCompile` exists and looks like the correct package. Using existing path.
  Resolving package versions...
ERROR: Unsatisfiable requirements detected for package SnoopCompileBot [1d5e0e55]:
 SnoopCompileBot [1d5e0e55] log:
 ├─possible versions are: 1.6.0 or uninstalled
 └─restricted to versions 1.6.1-1.6 by SnoopCompile [aa65fe97] — no versions left
   └─SnoopCompile [aa65fe97] log:
     ├─possible versions are: 1.6.1 or uninstalled
     └─SnoopCompile [aa65fe97] is fixed to version 1.6.1

(@v1.6) pkg> resolve
No Changes to `~/.julia/environments/v1.6/Project.toml`
No Changes to `~/.julia/environments/v1.6/Manifest.toml`

(@v1.6) pkg> dev SnoopCompile
Path `/home/tim/.julia/dev/SnoopCompile` exists and looks like the correct package. Using existing path.
  Resolving package versions...
ERROR: Unsatisfiable requirements detected for package SnoopCompileBot [1d5e0e55]:
 SnoopCompileBot [1d5e0e55] log:
 ├─possible versions are: 1.6.0 or uninstalled
 └─restricted to versions 1.6.1-1.6 by SnoopCompile [aa65fe97] — no versions left
   └─SnoopCompile [aa65fe97] log:
     ├─possible versions are: 1.6.1 or uninstalled
     └─SnoopCompile [aa65fe97] is fixed to version 1.6.1

(@v1.6) pkg> dev SnoopCompile SnoopCompileCore SnoopCompileAnalysis SnoopCompileBot
[ Info: Resolving package identifier `SnoopCompileCore` as a directory at `~/.julia/dev/SnoopCompile/SnoopCompileCore`.
[ Info: Resolving package identifier `SnoopCompileAnalysis` as a directory at `~/.julia/dev/SnoopCompile/SnoopCompileAnalysis`.
[ Info: Resolving package identifier `SnoopCompileBot` as a directory at `~/.julia/dev/SnoopCompile/SnoopCompileBot`.
Path `/home/tim/.julia/dev/SnoopCompile` exists and looks like the correct package. Using existing path.
Path `SnoopCompileCore` exists and looks like the correct package. Using existing path.
Path `SnoopCompileAnalysis` exists and looks like the correct package. Using existing path.
Path `SnoopCompileBot` exists and looks like the correct package. Using existing path.
  Resolving package versions...
Updating `~/.julia/environments/v1.6/Project.toml`
  [aa65fe97] ~ SnoopCompile v1.6.0 ⇒ v1.6.1 `~/.julia/dev/SnoopCompile`
  [9ea4277c] ~ SnoopCompileAnalysis v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileAnalysis`
  [1d5e0e55] ~ SnoopCompileBot v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileBot`
  [e2b509da] ~ SnoopCompileCore v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileCore`
Updating `~/.julia/environments/v1.6/Manifest.toml`
  [aa65fe97] ~ SnoopCompile v1.6.0 ⇒ v1.6.1 `~/.julia/dev/SnoopCompile`
  [9ea4277c] ~ SnoopCompileAnalysis v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileAnalysis`
  [1d5e0e55] ~ SnoopCompileBot v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileBot`
  [e2b509da] ~ SnoopCompileCore v1.6.0 ⇒ v1.6.1 `../../dev/SnoopCompile/SnoopCompileCore`

I'm not sure how it would be implemented, but the above is not ideal.

[johnnychen94]

Am I understand it wrongly, or a Pkg.update() seems already a good solution to this?

- JULIA_DEPOT_PATH=/tmp/pkgs julia-latest --startup-file=no --project -e 'using Pkg; Pkg.test("BigPkg", coverage=false)'
+ JULIA_DEPOT_PATH=/tmp/pkgs julia-latest --startup-file=no --project -e 'using Pkg; Pkg.update(); Pkg.test("BigPkg", coverage=false)'

jc@mathlf1:/tmp/pkgs/dev/BigPkg$ J ULIA_DEPOT_PATH=/tmp/pkgs julia-latest --startup-file=no --project -e 'using Pkg; Pkg.update(); Pkg.test("BigPkg", coverage=false)'

   Updating registry at `/tmp/pkgs/registries/General`
   Updating registry at `/tmp/pkgs/registries/MyRegistry`
   Updating git-repo `/tmp/pkgs/registries/MyRegistry`
   Updating git-repo `/tmp/pkgs/dev/BigPkg`
Updating `/tmp/pkgs/dev/BigPkg/Project.toml`
  [352b57a5] ~ SubPkg v0.1.0 `.:SubPkg#master` ⇒ v0.2.0 `.:SubPkg#master`
Updating `/tmp/pkgs/dev/BigPkg/Manifest.toml`
  [352b57a5] ~ SubPkg v0.1.0 `.:SubPkg#master` ⇒ v0.2.0 `.:SubPkg#master`
    Testing BigPkg
Status `/tmp/jl_zZBmcy/Project.toml`
  [5abf6831] BigPkg v0.2.0 `/tmp/pkgs/dev/BigPkg`
  [352b57a5] SubPkg v0.2.0 `.:SubPkg#master`
Status `/tmp/jl_zZBmcy/Manifest.toml`
  [5abf6831] BigPkg v0.2.0 `/tmp/pkgs/dev/BigPkg`
  [352b57a5] SubPkg v0.2.0 `.:SubPkg#master`
ERROR: LoadError: ArgumentError: Package Test not found in current path:
- Run `import Pkg; Pkg.add("Test")` to install the Test package.

Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:893
 [2] include(::String) at ./client.jl:444
 [3] top-level scope at none:6
in expression starting at /tmp/pkgs/dev/BigPkg/test/runtests.jl:1
ERROR: Package BigPkg errored during testing
Stacktrace:
 [1] pkgerror(::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:52
 [2] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Operations.jl:1561
 [3] test(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:327
 [4] #test#61 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:67 [inlined]
 [5] #test#60 at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:66 [inlined]
 [6] test(::String; kwargs::Base.Iterators.Pairs{Symbol,Bool,Tuple{Symbol},NamedTuple{(:coverage,),Tuple{Bool}}}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:65

[KristofferC]

I'm not sure how it would be implemented, but the above is not ideal.

What's the not ideal part? That you have to dev them in one command?

[johnnychen94]

My understanding is that when you dev a package, it should automatically dev all sub-packages since they're in one big git repo. It doesn't make much sense to still use the released version of those sub-packages.

Take SnoopCompile as the very specific case:

• (@v1.6) pkg> dev SnoopCompile: also devs SnoopCompile, SnoopCompileCore, SnoopCompileAnalysis, SnoopCompileBot
• (@v1.6) pkg> dev SnoopCompileBot: also devs SnoopCompileCore and SnoopCompileAnalysis
• (@v1.6) pkg> dev SnoopCompileCore: as it is

[aminya]

Yes. I made a dev.jl for my personal use to do that.
https://github.com/aminya/SnoopCompile.jl/blob/devscript/dev.jl

[KristofferC]

My understanding is that when you dev a package, it should automatically dev all sub-packages since they're in one big git repo. It doesn't make much sense to still use the released version of those sub-packages.

Just to point out, there is no concept (right now at least) of a sub-package. There are just packages and they can now be stored in the same git-repo. It is not clear at all to me that automatically devving all packages that are in the same repo just because you dev one of them is something you would always want.

Yes. I made a dev.jl for my personal use to do that.

Would be better to do all the devs in one command (develop can take a vector of PackageSpec).

[timholy]

Deciding specifically what is wrong is definitely a gray zone. Let me try in several steps:

• nothing is documented about this, everyone has to figure it out on their own. Various fixes (mostly for the split) timholy/SnoopCompile.jl#107 did not exactly happen automatically without effort. At a bare minimum this needs to be documented. I've edited the top post here to reflect that, but if that's all we end up doing then it needs to go somewhere more permanent.
• it's not completely intuitive. It might be to people who wrote Pkg (I am eternally grateful), but I think many of us want to think of one repo as an "uber-package." Really, the feature I think I really want (but don't have) is the ability to load specific sub-modules, in which case this really would be just one package. But that doesn't seem to be on the horizon (and it's not entirely trivial to see how it would work), and this already is 98% working, so I'm interested in exploring whether we can smooth over that last 2%. Yes, that last 2% might be a corruption of Pkg's design. I think we need to think about it anyway---the current workflow for large ecosystems is too painful. (That's not a new problem, but now we almost have a fix.)

It seems to me that we need some way of saying, "this collection of packages belongs together." Maybe something like a new

[subpkgs]
SubPkgA
SubPkgB
SubPkgC/SubPkgCa
SubPkgC/SubPkgCb

section to the "umbrella" Project.toml. Pkg looks for that and then does some stuff differently.

But I'm open to other solutions. I definitely don't want to create src-chaos if there's an easier way to smooth the workflow.

[timholy]

It is not clear at all to me that automatically devving all packages that are in the same repo just because you dev one of them is something you would always want.

I strongly agree with you there. This should be something you opt into. I'm not certain how flexible this needs to be? Should it be "all one or none"? Or would people want to group some packages but not others? E.g.,

[subpkgs]
SnoopCompile = ["SubPkgB", "SubPkgC/SubPkgCa", "SubPkgC/SubPkgCb"]
SomeOtherTool = ["ItsSubPkgA", ...]

[KristofferC]

It should be stated that the more that HugePackge gets cut up, the slower it is likely to load (ref JuliaGraphics/Gtk.jl#466).

[timholy]

Warning noted. But again, in the case I really care about, I've already done the cutting up (all the way back in the Julia 0.x for small x days). I want to glue them back together.

[johnnychen94]

Does this mean Minipackage doesn't need to be registered in the General registry anymore? If so there could be some confusion between using Minipackage and using HugePackage@minipackage, especially when the original Minipackage repo is deprecated and not updated.

How about package tests? What would happen when ] test HugePackage, does it include all tests in those Minipackages? will ] test HugePackage@Minipackage work?

[GunnarFarneback]

This was a too long read to go into all details, so please excuse me if I've missed or misunderstood something important.

I think the layout

BigPkg/
├── A
│   └── Project.toml
├── B
│   └── Project.toml
├── C
│   └── Project.toml
└── Project.toml

where BigPkg is an umbrella package for A, B, C, is the wrong model for this. I think it should rather be

Repo
├── A
│   └── Project.toml
├── B
│   └── Project.toml
├── BigPkg
│   └── Project.toml
├── C
│   └── Project.toml
├── Manifest.toml
└── Project.toml

where the top level Project.toml and Manifest.toml permanently devs all the packages in the repo. Then you would run CI from the top level repo and if it's fine, bump the package versions and register them all together. Obviously there's bit of tooling missing for the latter steps but I would be happy to host experiments on this in LocalRegistry, which would be a more convenient playground than Registrator.

(I have no personal need of this. My interest in subdir packages has always been on the case where the Julia package is only a smaller part of a repo.)

[ma-laforge]

It is not clear at all to me that automatically devving all packages that are in the same repo just because you dev one of them is something you would always want.

I strongly agree with you there. This should be something you opt into. I'm not certain how flexible this needs to be? Should it be "all one or none"? Or would people want to group some packages but not others? E.g.,

[subpkgs]
SnoopCompile = ["SubPkgB", "SubPkgC/SubPkgCa", "SubPkgC/SubPkgCb"]
SomeOtherTool = ["ItsSubPkgA", ...]

I think I disagree with you here, @KristofferC & @timholy. If you create a multi-package repo in a single .git repository, I think this sort of makes sense.

dev-ing "affects" all sub-packages

On a purely practical standpoint, you must check out the entire repo when you try to dev one of the subpackages. So really, I think dev RootPkg should trigger devving of all sub-packages in that repo -- and have all subsequent calls to import RootPkg@SubPkg* redirect to the dev-ed version of the package.

It is the most natural, orthogonal way to do development, I think.

Repo as an organizational unit

In most cases, package developers should use the .git repo as a means to collect somewhat related packages/functionality only.

If we want to support the possibility for developers dump all kinds of unrelated packages in a single repo... well, that's fine. However, I really think the primary goal should be to have a simple & efficient workflow to deal with the former case, though.

IMPORTANT: I wish to promote the concept of sub-packages instead of using our present solution, as is. If we can develop a workflow with the sub-package concept in mind, I believe it can vastly improve the development of complex multi-package solutions within a single .git repo.

Use case: adding a new feature

Let's assume we bundled Plots.jl and its supporting packages into a single .git repo. When I eventually want to add a new feature to Plots, there is a good chance I'll also have to add code to to RecipesPipeline & RecipesBase in order to address the solution correctly.

#Plots.jl.git repo could contain all its related dependencies!:
Plots -> RecipesPipeline -> RecipesBase
         PlotUtils
         PlotThemes -> PlotUtils
         RecipesBase
         RecipesPipeline -> RecipesBase
                            PlotUtils

So yes: I think it makes tons of sense to have all sub-packages deved with a call to dev Plots.

Package-level micro-registries

To me, it makes the most sense to have package-level micro-registries. In theory, you should only need to register the root package (RootPkg) with JuliaRegistries/General. When you register (RootPkg), the registration tools should be able to parse the its Project.toml file in order to find all sub-packages:

name = "RootPkg"
uuid = "aa65fe97-06da-5843-b5b1-d5d13cad87d2"
version = "1.7.1"

[subpkgs]
SubPkgB = "subpkgs/SubPkgB"
SubPkgCa = "subpkgs/SubPkgC/SubPkgCa"
SubPkgCb = "subpkgs/SubPkgC/SubPkgCb"

[deps]
SubPkgB = "9ea4277c-da97-4c3a-afb0-537c066769de"
SubPkgCa = "1d5e0e55-7d74-4714-b8d8-efa80e938cf7"
SubPkgCb = "e2b509da-e806-4183-be48-004708413034"

Keeping versions together
What is key here is that these sub-packages should not typically need to be versioned In the sub-package-capable workflow. They should basically inherit the version number from RootPkg.

[ma-laforge]

@KristofferC What we need:

- [...]
- A file layout convention, let's say we make it: `src/MainPackage.jl` and `src/SubPackage.jl`.
- [...]

I like the simplicity of this file layout, but this might not be a practical way to declare the dependencies for MainPackage@SubPackage. If we instead place sub-packages in their own sub-directories, we can bundle in their own Project.toml file - thus allowing the sub-packages to have fewer dependencies.

Subpackages with MORE dependencies

If I were to add another requirement: it would be nice if certain subpackages could also have more dependencies than the root package. For example:

SnoopCompile [dep: Pkg, SnoopCompile@SnoopCompileCore, SnoopCompile@SnoopCompileAnalysis, ...]
    ->SnoopCompileCore [dep: Serialization]
    ->SnoopCompileAnalysis [dep: Cthulhu, OrderedCollections, Serialization]
    ->SnoopCompileBot [dep: Pkg, FilePathsBase, SnoopCompile@SnoopCompileCore, ...]

#Additional/Wrapper packages:
    ->SnoopCompileGUI [dep: SnoopCompile, Gtk, ...]

In the future, it might make sense to bundle a GUI to aid in using SnoopCompile. If we include it as a sub-project with additional dependencies, it means we don't have to add Gtk.jl to all Julia environments that want to use SnoopCompile.

Well, maybe a GUI for SnoopCompile wouldn't really be appropriate, but I hope you get the idea.

[ma-laforge]

@GunnarFarneback where BigPkg is an umbrella package for A, B, C, is the wrong model for this. I think it should rather be

Repo
├── A
│   └── Project.toml
├── B
│   └── Project.toml
├── BigPkg
│   └── Project.toml
├── C
│   └── Project.toml
├── Manifest.toml
└── Project.toml

Before answering, I would like to rename Repo->RootPkg to clarify the discussion. That way, you can have BOTH a RootPkg, and a BigPkg.

What I like about this topology is that RootPkg.Project.toml can be created have zero dependencies on its own. In turn, that would mean that:

• ] add RootPkg@A could pull in ONLY the direct dependencies of A.
• Most people will probably ] add RootPkg@BigPkg: Get all functionality from BigPkg.

I really like this idea. The only problem is that it is slightly less intuitive than @timholy 's original proposal where RootPkg=BigPkg. But that's just because you no longer get the implicit positional hint that all subpackages provide the functionality of RootPkg=BigPkg. Despite this small drawback, I think this solution is more flexible and appropriate.

[GunnarFarneback]

Before answering, I would like to rename Repo->RootPkg

I intentionally chose a non-package name for the root of the repo because

1. I'm not convinced that there really is a need for the subpackage concept. It looks more like a tooling problem to me.
2. Unless there is a major redesign of Pkg, a root package will contain the contents of all packages in subdirectories, as well as non-package subdirectories if there are any. I'm not sure everybody is aware of this.

[ma-laforge]

• I'm not convinced that there really is a need for the subpackage concept. It looks more like a tooling problem to me.

I'm not sure I understand. Yes, I think it is a tooling problem: the package/registration system lacks the ability to deal with multiple packages in the same .git repo in a way that is practical for development, testing, and keeping dependencies to a minimum. The subpackage concept is simply an abstraction we use to describe our problem/solution.

To me, this feels alot like the one file per function paradigm in Matlab. Yes, you can just bite your tongue and break everything into singe files, but you tend to get lost in your files, and development time suffers.

Actually, it is worse than the one file per function issue, because it causes long delays in creating releases. Even the part of "push" ing each individual .git repo/julia package to Github is a pain when your solution is broken down into so many repositories.

• Unless there is a major redesign of Pkg, a root package will contain the contents of all packages in subdirectories, as well as non-package subdirectories if there are any. I'm not sure everybody is aware of this.

Yes. A package was originally defined at the .git repo level. So yes, when you add that package, Julia pulls the entire repo. I assume most people know this. The decision to tie packages to a .git repo seems to be the core reason why there is a need for the sub-package concept.

And I agree, this might require a relatively major redesign of Pkg. If it can be done in an effective way by adding support for subdirectories, as well as a few registration scripts, then that would be excellent! Sadly, I'm not convinced that's all that we need to make this aspect of package development user-friendly.

[GunnarFarneback]

And I agree, this might require a relatively major redesign of Pkg

Well, that's where we disagree. I don't think Pkg needs (more) changes to effectively support multiple packages in one repository. I just think you shouldn't try to place a package at the root of the repository if it includes multiple packages.

[DilumAluthge]

Yeah, when I first suggested this feature, I did not imagine that one package would be in a subdirectory of another.

[ma-laforge]

Well, that's where we disagree. I don't think Pkg needs (more) changes to effectively support multiple packages in one repository.

If "effectively" is the operative word here, I think the difficulty @timholy is having to get SnoopCompile working in a multi-package context says otherwise. Maybe Pkg doesn't need more changes, but something probably does. I just don't know what that thing is at this time.

I just think you shouldn't try to place a package at the root of the repository if it includes multiple packages.

I have no strong opinion on this. In this context, I'm completely fine with there not being a package at the root of my repositories. This restriction doesn't really pose an obstacle for my particular use cases. In fact, it might end up being less confusing to add this restriction.

[johnnychen94]

The simplest package development workflow is to have one big package for all related functions. As @ma-laforge said, developing one big repo makes version control less a headache, especially when the change involves multiple packages. A gripe is that this giant package makes precompilation time a big issue for both developers and users.

I wanted to echo @timholy that the concept of sub-packages, IMO, gives us the best of two worlds that 1) it's easier to make a change to multiple sub-packages without propagating incompleteness to downstream packages 2) it doesn't explode the precompilation time.

[KristofferC]

FWIW, I am very much losing the overview of this issue, what is proposed, what is considered bug reports, what is considered feature requests, what is just random discussion from the SnoopCompile split etc. I'm going to close this, so please open a new issue with a concrete descriptive proposal.

[GunnarFarneback]

Let's go back to the beginning and apply the layout from #1874 (comment) to the reproducer. Then the generate_test.jl would be changed to

function writelines(path, lines)
    open(path, "w") do io
        for line in lines
            println(io, line)
        end
    end
end

# Create a Root repository with subdir packages BigPkg and SubPkg
# Make BigPkg depend on SubPkg
# Register them
using Pkg
Pkg.add("LocalRegistry")
using LocalRegistry
regpath = joinpath(@__DIR__, "registries", "MyRegistry")
create_registry("MyRegistry", regpath)
# Pkg.Registry.add(regpath)
devpath = joinpath(@__DIR__, "dev")
mkpath(devpath)
rootpath = joinpath(devpath, "Root")
bigpkgpath = joinpath(rootpath, "BigPkg")
subpkgpath = joinpath(rootpath, "SubPkg")
Pkg.generate(bigpkgpath)
Pkg.generate(subpkgpath)
Pkg.activate(rootpath)
Pkg.develop(path=bigpkgpath)
Pkg.develop(path=subpkgpath)
cd(rootpath) do
    run(`git init`)
    run(`git add Project.toml Manifest.toml BigPkg/Project.toml BigPkg/src/BigPkg.jl SubPkg/Project.toml SubPkg/src/SubPkg.jl`)
    run(`git commit -m "Skeleton"`)

    Pkg.activate("BigPkg")
    Pkg.add("Test")
    Pkg.add(path=".", subdir="SubPkg")
    lines = readlines("BigPkg/Project.toml")
    append!(lines, ["", "[compat]", "SubPkg = \"0.1\""])
    writelines("BigPkg/Project.toml", lines)
    lines = readlines("BigPkg/src/BigPkg.jl")
    insert!(lines, 2, "using SubPkg")
    writelines("BigPkg/src/BigPkg.jl", lines)
    Pkg.activate(rootpath)
    Pkg.resolve()
    run(`git add BigPkg/Project.toml BigPkg/src/BigPkg.jl`)
    run(`git commit -m "Add SubPkg dependency"`)
end
using BigPkg
using SubPkg
register(SubPkg, "MyRegistry"; repo=bigpkgpath)
register(BigPkg, "MyRegistry"; repo=bigpkgpath)

# OK, now both packages are registered at v0.1.
# Let's add a new feature to SubPkg and bump its version
# to 0.2. Simultaneously, let's require the new feature
# in BigPkg and bump its version to 0.2.
cd(subpkgpath) do
    lines = readlines("src/SubPkg.jl")
    insert!(lines, 2, "export f")
    insert!(lines, 3, "f() = 1")
    writelines("src/SubPkg.jl", lines)
    lines = readlines("Project.toml")
    idx = findfirst(str->startswith(str, "version"), lines)
    lines[idx] = "version = \"0.2.0\""
    writelines("Project.toml", lines)
end
cd(bigpkgpath) do
    lines = readlines("src/BigPkg.jl")
    insert!(lines, 3, "g() = f()")
    writelines("src/BigPkg.jl", lines)
    lines = readlines("Project.toml")
    idx = findfirst(str->startswith(str, "version"), lines)
    lines[idx] = "version = \"0.2.0\""
    lines[end] = "SubPkg = \"0.2\""
    writelines("Project.toml", lines)
    mkdir("test")
    open("test/runtests.jl", "w") do io
        println(io, """
        using BigPkg, Test
        @test BigPkg.g() == 1
        """)
    end

    run(`git add Project.toml src/BigPkg.jl ../SubPkg/Project.toml ../SubPkg/src/SubPkg.jl test/runtests.jl`)
    run(`git commit -m "Version 0.2"`)
end

Performing the same steps as in the issue description, I get

gunnar@ubuntu:/tmp/pkgs/dev/Root$ JULIA_DEPOT_PATH=/tmp/pkgs ~/julia1.5/julia --startup-file=no --project -e 'using Pkg; Pkg.test("BigPkg", coverage=false)'
    Testing BigPkg
Status `/tmp/jl_9a76kQ/Project.toml`
  [5f1f962e] BigPkg v0.2.0 `/tmp/pkgs/dev/Root/BigPkg`
  [d729db4a] SubPkg v0.2.0 `/tmp/pkgs/dev/Root/SubPkg`
  [8dfed614] Test
Status `/tmp/jl_9a76kQ/Manifest.toml`
  [5f1f962e] BigPkg v0.2.0 `/tmp/pkgs/dev/Root/BigPkg`
  [d729db4a] SubPkg v0.2.0 `/tmp/pkgs/dev/Root/SubPkg`
  [2a0f44e3] Base64
  [8ba89e20] Distributed
  [b77e0a4c] InteractiveUtils
  [56ddb016] Logging
  [d6f4376e] Markdown
  [9a3f8284] Random
  [9e88b42a] Serialization
  [6462fe0b] Sockets
  [8dfed614] Test
    Testing BigPkg tests passed 

@timholy, is this enough to solve this part of the puzzle?

[StefanKarpinski]

With apologies to @timholy, when this got to "relatively major redesign of Pkg" I think we lost the thread here. Let's try again with some more focused issues. If folks want to continue the larger brainstorming here, feel free.

[ma-laforge]

I apologize as well. When I said "this might require a relatively major redesign of Pkg", I actually meant I expect the changes won't be a couple of non-trivial one liners.

Dealing with multi-package repositories in a way that is easy for development, unit testing, and publication/registration requires a bit of thought. This is especially true if we want to maintain minimum sub-package dependencies to avoid pulling in more packages than a project requires.

However, for the most part, I think the package system does an excellent job dealing with very complex issues.

[StefanKarpinski]

This issue has come up again on Slack and the fact that I "hated it" was mentioned as the reason for abandoning the idea. To be clear: I would be totally fine with a feature that allowed import A.B to load B without having to load all of the rest of A. However, such a feature should

1. not introduce any new concepts that the caller needs to understand
2. not introduce any new syntax so that the caller can just do import A.B and not be aware that B is special

The proposals being discussed here violate both of those and introduces a subtly different new concept—so subtle that from the user's perspective it's identical to a submodule, it only different from the developer's perspective, which seems not great, Bob. It also introduces a new syntax which, so that the user has to guess which of import A.B or import A@B they're supposed to write. Since there's no user-visible difference in the effect of these, it's a pure guessing game which one they're supposed to write.

[KristofferC]

The reason for the new syntax is only to be backward compatible with the current semantics of A.B (which is to load all of A). Let's put import A.B is not guaranteed to load A to the 2.0 breaking change list? :P

[StefanKarpinski]

If the package opts into it, then changing the behavior of import A.B would be acceptable. If that stands a chance of causing a breaking change to callers, then that's on the package developer.

[KristofferC]

Yeah, that's probably a better version of opting into it than the caller.

[StefanKarpinski]

There are several counter-arguments to the need for this feature in the first place:

1. If you can import B without importing A and B is independent enough that it makes sense to do so, then B should be separate from A anyway.

2. It significantly overlaps with the concept of package namespaces if you just say that import A@B is the syntax for loading package B from namespace A, which has had import A/B proposed as a syntax. How many of these slightly different ways to import a package do we really want? Arguably with import and using we already have too many that are too subtly different.

Again: this does not mean that I reject the problem, just some shapes of solutions. Introducing a package namespace concept is slightly more palatable since it's a fairly clear concept, as opposed to subpackages, which seems a bit contradictory: "a package is a unit of reusable code"; "a subpackage is an even smaller unit of reusable code". From the namespace perspective, if we see import A.B and we know that A is a namespace, then we can load B as a package. That's just a code loading feature. The same sort of thing could be done for "subpackages", but again, I don't think we should introduce two such similar features unless we can figure out one design that solves both problems.

[KristofferC]

I wouldn't get too stuck on the particular names I picked for everything. To me, it doesn't matter if it is called a "subpackage", a package B namespaced by A, a conditional part of package A etc. The question is, what does import A.B actually do (assuming the package opted in to it) and what it can be used for. In this proposal, it loads src/B.jl and precompiles that into its own .ji file using the same Project + Manifest as A would have used. What it can be used for is e.g. conditional loading of dependencies where import A.GPU gives you GPU support but is not loaded when you just du import A.

I don't see how it overlaps with the package namespace proposal at all. That is only about how one can disambiguate the string -> UUID lookup that we have. So instead of only having add A resolve to the UUID with name A in the registry you could have add NameSpace/A which finds the UUID of that some other way. It doesn't do anything with code loading which this proposal is fundamentally about. Basically, the namespace proposal will only touch the code in the package manager and registry, this proposal will only touch code in code loading.

[StefanKarpinski]

The overlap is that they both introduce import A.B (where the . might be a different symbol) for which B is the package and A is something that is not a package but some kind of collection of packages which does not need to be loaded. Focusing on the different mechanics of the two is missing the point that there's a conceptual commonality here and it's kind of questionable to introduce two different mechanisms for accomplishing similar things. If I want import A.B to only load code for B but not A I have two different options: make B a subpackage of A or make B a package in the namespace A. Which should I do?

[ma-laforge]

Sorry. I didn't notice you replied.

Package namespaces. Yes. That sounds great to me. I don't really need the subpackage concept. I understand that might end up causing subtle issues. That was not the intent.

Need:

• What I want is a library of packages that can be bundled in a single .git repository. Package namespaces sound like they could do exactly what I need!
• I want to be able to publish a bunch of packages in a "library" (package namespace) simultaneously. I would like to simplify the process of publishing a "library" of packages for solutions where more than one of these packages often need to be in sync for the "client software" to work properly.
• It would be nice if I only had to register a single package namespace in the Julia registry. After that, only select packages of that namespace could be added to the active environment, depending on its needs.
• It would also be nice if all the packages in that namespace were versioned together (so, I guess the version would belong to the namespace). I think that if developers want individual packages to have different versions in a namespace, well they probably shouldn't be bundling them as a single library.