Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PyTorch easyblock fails to properly count total number of test errors #2802

Closed
casparvl opened this issue Oct 14, 2022 · 0 comments · Fixed by #2803
Closed

PyTorch easyblock fails to properly count total number of test errors #2802

casparvl opened this issue Oct 14, 2022 · 0 comments · Fixed by #2803
Milestone

Comments

@casparvl
Copy link
Contributor

Just found out from a log posted in easybuilders/easybuild-easyconfigs#16385 that #2794 actually contains a mistake in the regular expressions, due to which the total number of tests errors is not counted correctly:

== 2022-10-12 02:20:36,991 pytorch.py:344 WARNING 0 test failure, 24 test errors (out of 88784):
distributed/pipeline/sync/skip/test_gpipe (12 skipped, 1 warning, 1 error)
distributed/pipeline/sync/skip/test_leak (1 warning, 8 errors)
distributed/pipeline/sync/test_bugs (1 skipped, 1 warning, 3 errors)
distributed/pipeline/sync/test_inplace (2 xfailed, 1 warning, 1 error)
distributed/pipeline/sync/test_pipe (1 passed, 8 skipped, 1 warning, 47 errors)
distributed/pipeline/sync/test_transparency (1 warning, 1 error)
distributions/test_distributions (216 total tests, errors=3, skipped=5)

Clearly, since one test suite already has 47 errors, the total can't be 24. Looking a bit close, the problematic regex is

^.*(?P<errors>[0-9]+) error.*$

Which on this line

distributed/pipeline/sync/test_pipe (1 passed, 8 skipped, 1 warning, 47 errors)

Only matches the 7 (the 4 is matched by the .*).

This can be fixed with

^.*[^0-9](?P<errors>[0-9]+) error.*$

I'll create a PR for that...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants