Add certified conformance mode and non-disruptive-conformance mode #885
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
With K8s v1.16, disruptive tests may not be part of the conformance
suite. As a result, we need to ensure that user workloads are
protected by default while still allowing for CNCF certification
test runs.
In this PR we:
conformance
mode to benon-disruptive-conformance
and modified the skip list appropriately.
certified-conformance
mode which does not provide a skip listand will run even the disruptive tests
consistent with our other flags/naming
Which issue(s) this PR fixes
Fixes #877
Fixes #875
Special notes for your reviewer:
I could have written the skip list in a slightly different way, naming the tests individually, but instead chose a higher level term that filtered both and, IMO, seems more clear why they may be skipped.
However, we need to talk about this issue in general because we are actually in a situation now where the
E2E_SKIP
value is dependent on the version of the cluster :( I think we can get away with this because the only real risks are:If there is a situation I've failed to think of, they can still set the e2e-focus/e2e-skip values on the CLI or the env vars in the YAML directly so we are not a blocker.
Release note: