[Feature] Skip prerequisite tasks

[wvwatson]

Is your feature request related to a problem? Please describe.

Currently there is no way to skip a dependency for the CNF conformance suite. It would be nice to be able to skip specific long running dependencies such as k8s_conformance at the command line if the user has already run that dependency using other methods.

Describe the solution you'd like

An example of skipping a dependency could work like this:

# skip k8s_conformance
./cnf-conformance all ~k8s_conformance

See #401 for the inspiration for this request

[wvwatson]

Related:
imdrasil/sam.cr/issues/15
imdrasil/sam.cr/pull/16
#372

[agentpoyo]

LGTM 👍

But it appears that it might still count the points in the skipped tasks/tests (my quick test to skip ~resilience ended up only getting around 30 points out of 105 total)? I know that makes sense overall but could be confusing for some as they might expect the tests that did run to show "passed" if they achieved enough points on given tests.

Would be nice if it showed a sum of points total on tasks that were actually run/not skipped but then also an overall points total that includes the points not counted in the skipped tasks (which then shows overall failure, since all tests should be run when checking conformance).

[lixuna]

@wavell did "skipping pre-req tasks" change how points are tallied?

[wvwatson]

"Would be nice if it showed a sum of points total on tasks that were actually run/not skipped but then also an overall points total that includes the points not counted in the skipped tasks (which then shows overall failure, since all tests should be run when checking conformance)."

Maybe would make for a good ticket w/planning. Not 100% sure on if reducing the amount of total points on skipped tasks is wanted though. Seems reasonable enough.

[agentpoyo]

"Would be nice if it showed a sum of points total on tasks that were actually run/not skipped but then also an overall points total that includes the points not counted in the skipped tasks (which then shows overall failure, since all tests should be run when checking conformance)."

Maybe would make for a good ticket w/planning. Not 100% sure on if reducing the amount of total points on skipped tasks is wanted though. Seems reasonable enough.

Yeah, in the real world, if you skip a test, you should still lose such points from the grand total but as an end user, it would nice to see a summary of the points of the actual tests run. I guess if end users are going to skip tests (or maybe the tasks/tests don't apply to their CNF and they feel they shouldn't be dinged on it), they should be aware their result will always likely show a "failed" status since their losing out on a handful or more points when skipping stuff.

[agentpoyo]

This still works I think as intended. As @wavell pointed out above, probably would need a new ticket with change of scope or behavior if skipping tests is to remove associated points with them from total tally.