Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix FreeBSD (incl. CI) reporting swallowing errors on reruns #13156

Merged
merged 1 commit into from
Feb 26, 2022

Conversation

nabijaczleweli
Copy link
Contributor

@nabijaczleweli nabijaczleweli commented Feb 26, 2022

Motivation and Context

Turns out, when your test-suite fails with

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create : No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create : No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create : No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create : No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create : No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create : No such file or directory

and succeeds, that's a bad thing!!

Description

Make it not do that. Hopefully. I'm delirious and there's a like a war or something next door.

How Has This Been Tested?

This has driven me crazy in #12996 (comment) as I tried to bisect it on CI, so if the FreeBSD CI fails then it's good.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Performance enhancement (non-breaking change which improves efficiency)
  • Code cleanup (non-breaking change which makes code smaller or more readable)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
  • Documentation (a change to man pages or other documentation)

Checklist:

  • My code follows the OpenZFS code style requirements.
  • I have updated the documentation accordingly.
  • I have read the contributing document.
  • I have added tests to cover my changes. – lol
  • I have run the ZFS Test Suite with this change applied. – ibid.
  • All commit messages are properly formatted and contain Signed-off-by.

@nabijaczleweli nabijaczleweli changed the title Fix freebsd reporting on reruns Fix FreeBSD (incl. CI) reporting on reruns Feb 26, 2022
@nabijaczleweli nabijaczleweli changed the title Fix FreeBSD (incl. CI) reporting on reruns Fix FreeBSD (incl. CI) reporting swallowing errors on reruns Feb 26, 2022
Turns out, when your test-suite fails with
Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
that's a bad thing!!

Fixes: 2320e6e ("Add zfs-test
 facility to automatically rerun failing tests")
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
@behlendorf behlendorf requested review from pcd1193182 and a user February 26, 2022 00:39
@behlendorf behlendorf added Component: Test Suite Indicates an issue with the test framework or a test case Status: Code Review Needed Ready for review and testing labels Feb 26, 2022
@nabijaczleweli
Copy link
Contributor Author

nabijaczleweli commented Feb 26, 2022

This passes, it's all green: http://build.zfsonlinux.org/builders/FreeBSD%20stable%2F13%20amd64%20%28TEST%29/builds/3961/steps/shell_4/logs/summary, it has 7 FAILs, among them receive-o-x_props_override, which is not ignored, and is caught correctly, e.g., here: http://build.zfsonlinux.org/builders/FreeBSD%20stable%2F13%20amd64%20%28TEST%29/builds/3968/steps/shell_4/logs/summary.

I don't actually know what broke it, because, well: FreeBSD CI was unreliable at best since 2320e6e. @behlendorf it is for this reason I'd like to (modestly, it's just CI, but I cannot possibly tell you how stark raving mad this has driven me over the past however-long I've been trying to even remotely understand why #12996 is failing in stuff I haven't vaguely touched) fast-track this so we do actually catch new test failures on FreeBSD.

Copy link
Contributor

@behlendorf behlendorf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's unfortunate. It seems we didn't catch this initially since the GitHub Actions builders only test Linux, and then when we enabled it 3 weeks ago for the other CI builders in openzfs/zfs-buildbot#243 we didn't notice the errors in the FreeBSD log. Let's just wait for the CI to finish with it.

rincebrain added a commit to rincebrain/zfs that referenced this pull request Feb 26, 2022
(Not worrying about largest_pool_001_pos or receive -o/-x on FBSD, because
those appear to be broken on vanilla and not noticed because openzfs#13156 isn't
landed yet.)

Signed-off-by: Rich Ercolani <[email protected]>
@behlendorf behlendorf added Status: Accepted Ready to integrate (reviewed, tested) and removed Status: Code Review Needed Ready for review and testing labels Feb 26, 2022
@behlendorf behlendorf merged commit 4f453dc into openzfs:master Feb 26, 2022
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Mar 1, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
rincebrain added a commit to rincebrain/zfs that referenced this pull request Mar 1, 2022
(Not worrying about largest_pool_001_pos or receive -o/-x on FBSD, because
those appear to be broken on vanilla and not noticed because openzfs#13156 isn't
landed yet.)

Signed-off-by: Rich Ercolani <[email protected]>
nicman23 pushed a commit to nicman23/zfs that referenced this pull request Aug 22, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
nicman23 pushed a commit to nicman23/zfs that referenced this pull request Aug 22, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
andrewc12 pushed a commit to andrewc12/openzfs that referenced this pull request Aug 30, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
lundman pushed a commit to openzfsonwindows/openzfs that referenced this pull request Sep 1, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
andrewc12 pushed a commit to andrewc12/openzfs that referenced this pull request Sep 23, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
andrewc12 pushed a commit to andrewc12/openzfs that referenced this pull request Sep 23, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
andrewc12 pushed a commit to andrewc12/openzfs that referenced this pull request Sep 23, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
andrewc12 pushed a commit to andrewc12/openzfs that referenced this pull request Sep 23, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
andrewc12 pushed a commit to andrewc12/openzfs that referenced this pull request Sep 23, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
andrewc12 pushed a commit to andrewc12/openzfs that referenced this pull request Sep 23, 2022
Turns out, when your test-suite fails on FreeBSD the rerun logic
would fail as follows:

Results Summary
PASS	 1358
FAIL	   7
SKIP	  47

Running Time:	04:00:02
Percent passed:	96.2%
Log directory:	/var/tmp/test_results/20220225T092538
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
mktemp: illegal option -- p
usage: mktemp [-d] [-q] [-t prefix] [-u] template ...
       mktemp [-d] [-q] [-u] -t prefix
/usr/local/share/zfs/zfs-tests.sh: cannot create :
                                   No such file or directory
...

This change resolves a flaw from the original commit, 2320e6e
("Add zfs-test  facility to automatically rerun failing tests")

Reviewed by: Brian Behlendorf <[email protected]>
Signed-off-by: Ahelenia Ziemiańska <[email protected]>
Closes openzfs#13156
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Component: Test Suite Indicates an issue with the test framework or a test case Status: Accepted Ready to integrate (reviewed, tested)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants