Enhance V2 test format to allow 0 priority assertions in tests.csv

[mcking65]

Currently, if a test includes an assertion that applies to only one of several commands, the assertion is listed in the assertions column of tests.csv and then it is removed from commands where it is not wanted by specifying a priority of 0 in the assertionsExceptions column of the *-commands.csv files. This creates the need for lots of exceptions when there is an assertion that is only applicable to one or two commands out of several and only applies to a subset of the in scope AT.

This proposal aims to simplify the csv file content by inverting the logic for such scenarios. If we could specify an assertion priority of 0 in tests.csv, that would make it possible to override the 0 priority with 1, 2, or 3 for only the commands where that assertion is applicable.

One might ask why is it necessary to specify the assertion in tests.csv at all. Why not simply add it where desired in the command.csv files. The answer is that the order of assertions in the presentation of assertions for a command is determined by the order of the assertionIDS in the assertions column of tests.csv. We could avoid the need to include the assertion in tests.csv in scenarios where it is desired for the exceptional additional assertion to be the last assertion in the presentation of the assertions for a command. That is, we could specify that if an assertion is not included in tests.csv but is included in commands.csv for a given test, then that assertion is appended to the list generated from tests.csv. If such logic is not too opaque, that would be a reasonable way to further simplify the csv files in some scenarios.

I've discovered there are also some scenarios where it would be helpful to replace a specific assertion with a different one for only some commands. For example, when navigating backward into a radio group where the first button in the group is checked, using shift+tab in reading mode moves to the first radio where as using up arrow moves the reading cursor to the last radio. This is expected. In this case, it would be helpful for the tests.csv to include assertions for both the first and last radio but with different priorities so only one is included. Then, in commands.csv, the shift+tab command could have exception that turn off assertions related to the third radio and turn on assertions related to the first.

In PR #1040, tests.csv and commands.csv illustrate these scenarios.

[mcking65]

@ccanash @howard-e

Please size and prioritize this ASAP. It provides a far superior path forward for the radio group test plan and test plans for any other similar composite widgets that require testing of mode switching and have similar challenges where shift+tab differs from other commands when navigating backward.

[ccanash]

Hi @mcking65 confirming this has entered our triaging process

[mcking65]

@ccanash @howard-e

I'd like to finalize and set a delivery date in today's CG meeting so we can build a schedule for refactoring the test plans that would need this change.

[howard-e]

#1042 should address this

[mcking65]

#1042 completed the scope agreed to for this issue. It is tested and merged. So, I am closing this issue.

There are two other ideas that have been proposed as part of this issue or during discussion of it. They are covered by #1045 and #1046.