Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: Improve spectest runner output #453

Merged
merged 2 commits into from
Aug 4, 2020
Merged

test: Improve spectest runner output #453

merged 2 commits into from
Aug 4, 2020

Conversation

gumb0
Copy link
Collaborator

@gumb0 gumb0 commented Aug 4, 2020 •
edited
Loading

Closes #450

Current output without skipped:

> bin/fizzy-spectests ~/dev/wasm-spec/test/core/json/float_memory/
Running tests from /home/andrei/dev/wasm-spec/test/core/json/float_memory/float_memory.json
..s..ssFs..s..s..s..ssFs..s..s..s..ssFs..s..s..s..ssFs..s..s..s..ssFs..s..s..s..ssFs..s..s
Line 21: assert_return FAILED Incorrect returned value. Expected: 2141192192 (0x7fa00000) Actual: 0 (0x0)
Line 46: assert_return FAILED Incorrect returned value. Expected: 9219994337134247936 (0x7ff4000000000000) Actual: 0 (0x0)
Line 73: assert_return FAILED Incorrect returned value. Expected: 2141192192 (0x7fa00000) Actual: 0 (0x0)
Line 98: assert_return FAILED Incorrect returned value. Expected: 9219994337134247936 (0x7ff4000000000000) Actual: 0 (0x0)
Line 125: assert_return FAILED Incorrect returned value. Expected: 2144337921 (0x7fd00001) Actual: 0 (0x0)
Line 150: assert_return FAILED Incorrect returned value. Expected: 9222246136947933185 (0x7ffc000000000001) Actual: 0 (0x0)
90 tests ran from float_memory.json.
  PASSED 48, FAILED 6, SKIPPED 36.

TOTAL 90 tests ran from "/home/andrei/dev/wasm-spec/test/core/json/float_memory/".
  PASSED 48, FAILED 6, SKIPPED 36.

With skipped:

> bin/fizzy-spectests ~/dev/wasm-spec/test/core/json/float_memory/ --show-skipped
Running tests from /home/andrei/dev/wasm-spec/test/core/json/float_memory/float_memory.json
..s..ssFs..s..s..s..ssFs..s..s..s..ssFs..s..s..s..ssFs..s..s..s..ssFs..s..s..s..ssFs..s..s
Line 16: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 19: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 20: action SKIPPED Unsupported feature: Floating point instruction.
Line 21: assert_return FAILED Incorrect returned value. Expected: 2141192192 (0x7fa00000) Actual: 0 (0x0)
Line 22: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 25: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 28: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 41: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 44: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 45: action SKIPPED Unsupported feature: Floating point instruction.
Line 46: assert_return FAILED Incorrect returned value. Expected: 9219994337134247936 (0x7ff4000000000000) Actual: 0 (0x0)
Line 47: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 50: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 53: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 68: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 71: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 72: action SKIPPED Unsupported feature: Floating point instruction.
Line 73: assert_return FAILED Incorrect returned value. Expected: 2141192192 (0x7fa00000) Actual: 0 (0x0)
Line 74: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 77: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 80: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 93: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 96: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 97: action SKIPPED Unsupported feature: Floating point instruction.
Line 98: assert_return FAILED Incorrect returned value. Expected: 9219994337134247936 (0x7ff4000000000000) Actual: 0 (0x0)
Line 99: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 102: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 105: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 120: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 123: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 124: action SKIPPED Unsupported feature: Floating point instruction.
Line 125: assert_return FAILED Incorrect returned value. Expected: 2144337921 (0x7fd00001) Actual: 0 (0x0)
Line 126: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 129: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 132: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 145: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 148: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 149: action SKIPPED Unsupported feature: Floating point instruction.
Line 150: assert_return FAILED Incorrect returned value. Expected: 9222246136947933185 (0x7ffc000000000001) Actual: 0 (0x0)
Line 151: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 154: assert_return SKIPPED Unsupported feature: Floating point instruction.
Line 157: assert_return SKIPPED Unsupported feature: Floating point instruction.
90 tests ran from float_memory.json.
  PASSED 48, FAILED 6, SKIPPED 36.

TOTAL 90 tests ran from "/home/andrei/dev/wasm-spec/test/core/json/float_memory/".
  PASSED 48, FAILED 6, SKIPPED 36.

Also supported are --show-passed and --hide-failed.

@gumb0 gumb0 force-pushed the spectest-verbosity branch from 4e05bc0 to e1807c7 Compare August 4, 2020 11:21
@gumb0 gumb0 marked this pull request as ready for review August 4, 2020 12:24
gumb0
gumb0 commented Aug 4, 2020
View reviewed changes
test/spectests/spectests.cpp

if (type == "module")
{
const auto filename = cmd.at("filename").get<std::string>();
log_no_newline("Instantiating " + filename + " ");
Copy link
Collaborator Author

@gumb0 gumb0 Aug 4, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing this, as I don't think it was very useful, and it would complicate things a bit to keep it.
The line will read just Line 3: module PASSED (instead of Line 3: module Instantiating call.0.wasm PASSED)

@codecov
Copy link

codecov bot commented Aug 4, 2020 •
edited
Loading

Codecov Report

Merging #453 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #453   +/-   ##
=======================================
  Coverage   99.50%   99.50%           
=======================================
  Files          52       52           
  Lines       14791    14813   +22     
=======================================
+ Hits        14718    14740   +22     
  Misses         73       73

@gumb0 gumb0 force-pushed the spectest-verbosity branch from e1807c7 to f8de506 Compare August 4, 2020 12:30
@chfast
Copy link
Collaborator

chfast commented Aug 4, 2020

Looks nice. Just I would use lowercase s for skipped as in Python.

@gumb0 gumb0 force-pushed the spectest-verbosity branch from f8de506 to 70812ff Compare August 4, 2020 12:40
@gumb0 gumb0 force-pushed the spectest-verbosity branch from 70812ff to 30c92cc Compare August 4, 2020 12:41
@gumb0
Copy link
Collaborator Author

gumb0 commented Aug 4, 2020

Looks nice. Just I would use lowercase s for skipped as in Python.

Changed.

@axic
Copy link
Member

axic commented Aug 4, 2020 •
edited
Loading

In order to not have reduced coverage, ctest should run spectests with the different combinations of parameters. I think for most cases expectation can remain the same simplified one we have.

@gumb0 gumb0 force-pushed the spectest-verbosity branch from d932845 to e1f1dc4 Compare August 4, 2020 13:11
@gumb0 gumb0 force-pushed the spectest-verbosity branch from e1f1dc4 to 04d3eef Compare August 4, 2020 13:21
@gumb0 gumb0 merged commit c4a67e8 into master Aug 4, 2020
@gumb0 gumb0 deleted the spectest-verbosity branch August 4, 2020 13:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chfast chfast chfast approved these changes

@axic axic axic approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Limit verbosity of fizzy-spectests
3 participants
@gumb0 @chfast @axic