Print more context for complex test conditions #46138
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Keno
force-pushed
the
kf/testsetlet
branch
3 times, most recently
from
5ff7ef6 to
d90db4b
Compare
vtjnash
reviewed
Jul 22, 2022
stdlib/Test/src/Test.jl
Outdated
| source, | ||
| message_only) | ||
| end | ||
| end | ||
|
|
||
| # Fallback constructor without `context` argument (added in Julia 1.9). Remove in Julia 2.0. |
There was a problem hiding this comment.
Suggested change
| # Fallback constructor without `context` argument (added in Julia 1.9). Remove in Julia 2.0. | |
| # Deprecated fallback constructor without `context` argument (added in Julia 1.9). Remove in Julia 2.0. |
(will make it easier to grep for later)
vtjnash
reviewed
Jul 22, 2022
stdlib/Test/src/Test.jl
Outdated
| source::LineNumberNode | ||
| message_only::Bool | ||
| function Fail(test_type::Symbol, orig_expr, data, value, source::LineNumberNode, message_only::Bool=false) | ||
| function Fail(test_type::Symbol, orig_expr, data, value, context, source::LineNumberNode, message_only::Bool=false) |
There was a problem hiding this comment.
If context is set to a LineNumberNode (let source=@__SOURCE__), it will call the wrong (the deprecated) function?
There was a problem hiding this comment.
I guess I'll remove the =false for now to make it explicit.
DilumAluthge
added
testsystem
On some failures, e.g. in the cmdlineargs failure in [1], printing the failing expression doesn't give us very much information, because what we're doing is just testing a small part of a larger structure, but to really diagnose the failure, we'd really want to see the whole structure (in this case the stderr output, not just the failure bool). This introduces an additional `@testset` feature that lets failures optionally include a context variable that gets printed on failure. Example: ``` julia> @testset let v=(1,2,3) @test v[1] == 1 @test v[2] == 3 end Test Failed at REPL[8]:3 Expression: v[2] == 3 Evaluated: 2 == 3 Context: v = (1, 2, 3) ERROR: There was an error during testing ``` The syntax here is `@testset let`, which was previously unused. The testset is transparent and failures/successes are passed through directly to the parent testset. In particular, using this kind of testset does not cause additional nesting in the default testset print. [1] https://buildkite.com/julialang/julia-master/builds/14160#01822311-84f0-467a-a8af-a5d751f2c6ab/448-728
This was referenced Jul 26, 2022
|
Should we add this to NEWS? |
ffucci
pushed a commit
to ffucci/julia
that referenced
this pull request
Aug 11, 2022
On some failures, e.g. in the cmdlineargs failure in [1], printing the failing expression doesn't give us very much information, because what we're doing is just testing a small part of a larger structure, but to really diagnose the failure, we'd really want to see the whole structure (in this case the stderr output, not just the failure bool). This introduces an additional `@testset` feature that lets failures optionally include a context variable that gets printed on failure. Example: ``` julia> @testset let v=(1,2,3) @test v[1] == 1 @test v[2] == 3 end Test Failed at REPL[8]:3 Expression: v[2] == 3 Evaluated: 2 == 3 Context: v = (1, 2, 3) ERROR: There was an error during testing ``` The syntax here is `@testset let`, which was previously unused. The testset is transparent and failures/successes are passed through directly to the parent testset. In particular, using this kind of testset does not cause additional nesting in the default testset print. [1] https://buildkite.com/julialang/julia-master/builds/14160#01822311-84f0-467a-a8af-a5d751f2c6ab/448-728
pcjentsch
pushed a commit
to pcjentsch/julia
that referenced
this pull request
Aug 18, 2022
On some failures, e.g. in the cmdlineargs failure in [1], printing the failing expression doesn't give us very much information, because what we're doing is just testing a small part of a larger structure, but to really diagnose the failure, we'd really want to see the whole structure (in this case the stderr output, not just the failure bool). This introduces an additional `@testset` feature that lets failures optionally include a context variable that gets printed on failure. Example: ``` julia> @testset let v=(1,2,3) @test v[1] == 1 @test v[2] == 3 end Test Failed at REPL[8]:3 Expression: v[2] == 3 Evaluated: 2 == 3 Context: v = (1, 2, 3) ERROR: There was an error during testing ``` The syntax here is `@testset let`, which was previously unused. The testset is transparent and failures/successes are passed through directly to the parent testset. In particular, using this kind of testset does not cause additional nesting in the default testset print. [1] https://buildkite.com/julialang/julia-master/builds/14160#01822311-84f0-467a-a8af-a5d751f2c6ab/448-728
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
On some failures, e.g. in the cmdlineargs failure in [1],
printing the failing expression doesn't give us very much
information, because what we're doing is just testing a
small part of a larger structure, but to really diagnose
the failure, we'd really want to see the whole structure
(in this case the stderr output, not just the failure bool).
This introduces an additional
@testsetfeature that letsfailures optionally include a context variable that gets printed
on failure. Example:
The syntax here is
@testset let, which was previously unused.The testset is transparent and failures/successes are passed
through directly to the parent testset. In particular, using
this kind of testset does not cause additional nesting in
the default testset print.
[1] https://buildkite.com/julialang/julia-master/builds/14160#01822311-84f0-467a-a8af-a5d751f2c6ab/448-728