Making test dependencies available at the REPL

[tkoolen]

My current workflow for quickly iterating on new code and the unit tests for that code together involves includeing the .jl file containing the tests in the REPL so as to benefit from Revise.jl speedups. (The .jl file has the relevant test code the nicely isolated in its own module and is normally included from runtests.jl).

Even though the test dependencies are recorded in the [extras] section of the Project.toml and in that sense 'known', the only way to support the above workflow seems to be to recreate the test dependency setup in a separate Project.toml file and to activate that environment. It would be very nice if it were easier to have the test dependencies available, not just while running the test task.

I was wondering, following the response to #727 for build dependencies and the currently preferred way of building documentation, would it make sense to switch to a setup where there's a separate Project.toml and optionally Manifest.toml in the test directory? That way, the test dependency setup is trivially available as just another environment.

[tkoolen]

I think the proposal in the last paragraph could also help with #480 by the way; instead of introducing more REPL mode syntax for adding test dependencies, one could just activate test/Project.toml and add the dependency as normal.

[timholy]

Making this easy would make me insanely joyous. I tend to just add the packages that it flags as missing when I do include("runtests.jl"), but that leads to a pretty bloated default environment.

[fredrikekre]

See https://github.com/fredrikekre/Sandbox.jl for a mockup of this (and for build dependencies) I put together some time ago. Also related to #624.

[tkoolen]

Re: Sandbox.jl, for the Project.toml in e.g. test, why not ] dev .., so that you can just do import Pkg; Pkg.activate(@__DIR__) in runtests.jl, but also have the complete environment available outside of runtests.jl (i.e., no Pkg.develop calls needed, as they would already have been 'incorporated' into test/Project.toml)?

Secondarily, what does creating new.toml files in a temporary directory buy you?

[fredrikekre]

why not ] dev ..

I didn't want to commit the manifest.

Secondarily, what does creating new.toml files in a temporary directory buy you?

If we need to resolve we don't modify the files in the repo.

[fredrikekre]

#989 was just merged btw, so feel free to alpha-test it!

[timholy]

@fredrikekre, I'm sure I could read and figure out the implications of #989, but is there a simple summary of what we should do?

[fredrikekre]

See #1031 😛

Basically, if there is a Project.toml in the test directory it will be used as the testing environment.

[fredrikekre]

For example:

$ tree .
.
├── Project.toml
├── src
│   └── Foo.jl
└── test
    ├── Manifest.toml
    ├── Project.toml
    └── runtests.jl

2 directories, 5 files

$ cat Project.toml 
name = "Foo"
uuid = "0e01d4e8-2b90-11e9-0e6a-a786d061c086"
version = "0.1.0"

$ cat test/Project.toml 
[deps]
Foo = "0e01d4e8-2b90-11e9-0e6a-a786d061c086"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"

$ cat test/runtests.jl 
using Test
@test true

[StefanKarpinski]

I have to say, the more I see it, the more I like this solution. All we need is the ability to generate multiple mutually compatible manifests (just that, sounds easy enough, right?).

[timholy]

So basically what you're saying is we should add a Project.toml to test/, and then when we want to run the tests we first ]activate ., right? Meanwhile that same file will be used when normally running the tests so we no longer need the [extras] section?

[tkoolen]

Thanks for the explanation. I also think this will be much nicer than the [extras] and test = [...] solution, in no small part because it (hopefully) requires fewer special cases (∝ opportunity for bugs) for downstream tooling like LanguageServer.jl and PackageCompiler.jl.

I do worry a little that .toml files then seem to be context-dependent. If I (perhaps naively) were to use the workflow in the issue description (don't use the test task, but just run julia --project in test and manually include runtests.jl to get Revise.jl speedups), wouldn't I be testing the latest released version of Foo.jl instead of the one in the parent directory? Or is there now a special check in place for that? This is why I like the dev .. solution, but I see @fredrikekre's point regarding not wanting commit a Manifest.toml.

[timholy]

Also generalizes nicely to a [build] target, see JuliaDebug/ASTInterpreter2.jl#37 (review)

[fredrikekre]

So basically what you're saying is we should add a Project.toml to test/, and then when we want to run the tests we first ]activate ., right?

Yea, but what did you mean with the . we activate here? The testing directory?

Pkg.test will implicitly dev the package itself onto the test/Project.toml env, but that is a no-op if you already have done that yourself. So the answer to the issue "Making test dependencies available at the REPL" is now activate test basically. So, Pkg.test is now roughly equivalent to julia --project=test test/runtests.jl (completely mirroring how most people now build docs: julia --project=docs docs/make.jl).

Meanwhile that same file will be used when normally running the tests so we no longer need the [extras] section?

Yea, see above. If there is a test/Project.toml file then [extras] section will be ignored.

I do worry a little that .toml files then seem to be context-dependent. If I (perhaps naively) were to use the workflow in the issue description (don't use the test task, but just run julia --project in test and manually include runtests.jl to get Revise.jl speedups), wouldn't I be testing the latest released version of Foo.jl instead of the one in the parent directory? Or is there now a special check in place for that? This is why I like the dev .. solution, but I see @fredrikekre's point regarding not wanting commit a Manifest.toml.

Well, as stated above, pkg.test will implicitly dev the correct path, so that will test the correct version. Even if you don't commit the test/Manifest.toml you can still make sure that you have devd the path in the local .gitignored manifest. From a clean clone you would have to `julia --project=test -e'using Pkg; Pkg.develop(PackageSpec(path = "."))' once though.

Also generalizes nicely to a [build] target, see JuliaDebug/ASTInterpreter2.jl#37 (review)

Yea, I prototyped that in https://github.com/fredrikekre/Sandbox.jl as well, and David implemented that in #989 (i.e. a deps/Project.toml will be used during build) but that is not really tested and could perhaps be problematic if we have to recursively resolve/build packages to build packages.

[KristofferC]

Closing as dup of #624