Modify processing of known failing tests #754
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently, when specifying a
known-failing
file, the tests that are listed in the file are matched against the list of test cases. If there are tests or patterns that do not match, an error is thrown.However, this causes an issue when trying to mark one of the gRPC tests as known-failing because the gRPC test names are modified via the
filterGRPCImplTestCases
function. Tests have(grpc server impl)
,(grpc client impl)
or(grpc impl)
added as the penultimate value before the test name, depending on how they are run. But since this is done after the known-failing tests are matched, it is impossible to mark one of these gRPC tests as failing.So, this PR modifies the calculation of the test names a bit by creating a list of all known test permutations and then using that to match against any known-failing tests listed.
Note that there is an example
known-failing
file and a test deliberately set up to fail just to see how this works in action. These two files will be removed from this PR before it is merged.To run with these changes, use the command: