Add zfs-test facility to automatically rerun failing tests

[pcd1193182]

Motivation and Context

This was a project proposed as part of the Quality theme for the hackthon for the 2021 OpenZFS Developer Summit. The idea is to improve the usability of the automated tests that get run when a PR is created by having failing tests automatically rerun in order to make flaky tests less impactful.

Description

This started a port of the Illumos commit 13365 Add option to testrunner to re-run just failed tests. That commit contains the basic infrastructure in test-runner.py to filter a list of tests down to only include ones that failed in some previous logfile.

There are changes to zts-report.py.in to print the list of tests on the maybe list, which are the flaky ones. There's also the option to do the report as usual, but change the return code to a different value if all failures were on the maybe or known failure lists.

There are changes to zfs-tests.sh, which are what actually triggers rerunning the failing tests. If the rerun flag is passed, and the report indicates that all test failures were known failures or flaky tests, then we isolate the failed flaky tests and rerun them.

There are also changes to the github workflows to use the new option. This is a slightly different interface than the one they elected to use in Illumos; they added a -l flag that allows a logfile to be passed when you rerun the zfs-tests command. I elected to go with this approach as it was more ergonomic for the automation use case, though the -l option can also be added if needed.

How Has This Been Tested?

Local runs of zfs-test did automatically rerun the failing tests when the option was provided, and behaved as usual when it was omitted.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[tonynguien]

Looks good to me. Good stuff!