enable regex-filtering of testsets based on their descriptions

[rfourquet]

The current solutions (that I know of) to run only a certain subset (typically of size one) of testsets in a project, consist either to comment out portions of the test files, or to copy-paste the testset at the REPL (which often doesn't work seamlessly, as you have to replicate the state in the REPL, e.g. define global variables, functions (like replshow() in Julia's "test/show.jl" file) etc.

This is not ideal, and filtering run testsets based on their description would go a long way to simplify this process, in many cases at least.

This PR uses a "JULIA_TESTSET_REGEX" environment variable: if set, only testsets matching this regex are run (note that this can be used to exclude certain patterns, with "negative lookahead" regexes). Usually, only toplevel testsets should be filtered, otherwise, filtering in only the following tests for example would be complicated:

@testset "integers" begin
    @testset "addition" begin
        @test 1 + 1 == 2
    end
    # more sub-testsets ...
end
# needs a regex like r"integers|addition| ...", but how to not catch "addition" from `@testset "floats"` ?

There is a catch: in the Julia repo for example, tests in a file are implicitly wrapped in a testset. So in this case, you only want the filtering to happen for testsets at nesting level 2 (toplevel is 1). So there is an other environment variable which contains a comma-separated list of levels (depths) at which filtering must happen, "JULIA_TESTSET_DEPTHS" , defaulting to "1".
(The name "JULIA_TESTSET_DEPTHS" is far from ideal, suggestions welcome).

[antoine-levitt]

How about each testset gets assigned a hierarchically defined label? Ie the above would be integers/addition. Then you can just filter based on that.

[rfourquet]

How about each testset gets assigned a hierarchically defined label? Ie the above would be integers/addition

Interesting idea! it would allow to get away with the ugly "JULIA_TESTSET_DEPTHS".

With the current state of Julia's own tests infrastructure, it wouldn't work as if no testsets are run in a file, then it errors. But I guess this could be fixed.

[rfourquet]

So the hierarchical approach makes it easy to filter for level one, e.g. using r"^integers" in this example, but not so for other levels. For example, to only run the tests integers/addition, you would have to use a regex like r"^integers$|^integers/addition", because integers testset needs to run first to give a chance to integers/addition to run... this is defintely doable, and probably not so needed in practice, but does someone see a simpler way?

[rfourquet]

Going down the rabbit hole: one solution may be to use PCRE's "partial matching" capabilities. If a testset's current description matches the regex r, then just run it. Otherwise, if it can be the prefix of a description which would match, then run it but disable the the @tests inside it (if possible), in case a sub-testset would match r.

[antoine-levitt]

I'm not sure if regexpes are the way to go here. Eg JuliaLang/Pkg.jl#1226 instead uses a list of tests to run, we could just do that without regexp? It seems to me that the common use case for the feature is "ok I changed that part of the code, I just need to run the corresponding test", for which passing a list of tests (in hierarchical format, but not regexp) would be the most user-friendly option. The good thing with regexp is that you can easily exclude specific tests ("I know that one will break, but are the others ok?"), but I would imagine that's less common? (Also I don't know for others, but I rarely use regexps, and every time I do it's in a different program and I have to look up the syntax over again, which is always annoying)

[rfourquet]

Pkg could eventually offer an option to pass a list of tests to run, but it can be implemented in different ways. It could be up to the package author to implement the logic in a runtests function (like is done in julia's tests), or it could rely on a mechanism provided by Test, like this PR proposes. I didn't spend time thinking about how Pkg can do it, but with the approach here, with an environment variable, I quite prefer regexes. How to specify the different tests to run with simple strings? it could be a list of comma separated names, e.g. "integers,integers/addition". But with regexes, just replace , with | and this works. So regex don't really complicate the simple case for the user. (But of course, as said in my previous message, it complicates the implementation if we want to support something like specifying only "integers/addition" and be sure the toplevel testset "integers" is run).

[bilderbuchi]

I hope it's ok to leave a drive-by comment. I am still inexperienced with Julia as today was the first time I've created some code and run it.

I come from Python, and in my opinion, the pytest test framework has done most (all?) things right and is powerful and a pleasure to use. I think you can't go wrong in taking design inspiration from pytest.
In relation to this thread, here is the section of the docs dealing with test selection when invoking pytest. I mainly find myself using the -k command line option.

[DilumAluthge]

Currently this PR uses the JULIA_TESTSET_REGEX environment variable.

Perhaps it could instead use ARGS, which would allow it to nicely integrate with the functionality in Pkg: JuliaLang/Pkg.jl#1226

[dpinol]

this will be a supercool feature. any plans to move it forward? thanks

[rfourquet]

I'm happy to move it forward if there is support, maybe it needs the triage label. In the meantime, at least two packages implemented filtering of testsets: https://github.com/RelationalAI-oss/XUnit.jl and https://github.com/JuliaTesting/ReTest.jl. For ReTest, I would suggest to wait a few days before trying it, until the next release.

[cossio]

This would be very nice to have. Any updates?

[JeffBezanson]

I much prefer the programmatic interface to this like ReTest has over environment variables. That would also make it easier to do fancy things like specify multi-component paths to match, e.g. ["integers", r"arith*"].

[senhalil]

Any updates on this topic? Currently we run our tests via julia --project=. -e 'using Pkg; Pkg.test()' and it is impossible to run aw single test.
It would be such a quality of life improvement if Pkg.test() had a regex argument to match testset names or filenames.