Revise reports to show assertion priorities as MUST/SHOULD/MAY instead of REQUIRED/OPTIONAL

[mcking65]

Problem

Currently, many important AT behaviors that are not absolutely essential to usage of a pattern are classified with the assertion priority "Required". Similarly, the assertion priority "optional" does not allow reports to distinguish between important behaviors that an AT should exhibit and potentially helpful but less important behaviors that an AT may exhibit.

These two characteristics of the current two-tier assertion prioritization system are having two significant impacts on the progress and utility of the ARIA-AT project:

1. They are impeding development of consensus.
2. They make reports less useful to consumers who are evaluating the suitability of a pattern for a specific purpose. This impact will be magnified when ARIA-AT starts evaluating new and emerging ARIA features.

Solution proposal

Align the assertion priorities with typical W3C normative requirement priorities of MUST, SHOULD, and MAY.

Implementation plan

In all existing test data:

• assertion priority 1 will be interpreted as "MUST" instead of "REQUIRED"
• assertion priority 2 will be interpreted as "SHOULD" instead of "OPTIONAL"

In the future, when test plans are converted to the V2 test format, some assertions will have a priority of 3, which maps to "MAY".

Assuming #733 is implemented, two aspects of data reporting need to change:

1. Display of priority data in command results tables.
2. Columns in report summary tables.

Changes to display of priority data in command results tables

In the tables specified by #733, there is a column for assertion priority. The data cells in that column need to show "MUST" for assertion priority 1 and "SHOULD" for assertion priority 2.

Changes to columns in report summary tables

In report tables that summarize results for each test using a specific AT and in a given browser:

• Change column heading "Required Assertions" to "MUST-DO Behaviors".
• Change column heading "Optional Assertions" to "SHOULD-DO Behaviors".
• Add a column "MAY-DO Behaviors".

[lolaodelola]

Hi @mcking65 I've added the agenda label so that we can discuss this further in the CG tomorrow. I understand it was discussed in a meeting while both Mike & I were away but I'd like to bring up some concerns we have as a group about the wording, particularly "MAY".

[css-meeting-bot]

The ARIA-AT Community Group just discussed New assertion priorities.

The full IRC log of that discussion

<jugglinmike> Topic: New assertion priorities
<jugglinmike> github: https://github.com//issues/737
<jugglinmike> Matt_King: Previously, we had "required" and "optional", but that didn't line up well with the W3C terminology of "MUST", "SHOULD", and "MAY"
<jugglinmike> Matt_King: The latest thinking is that we'll start by publishing a W3C note and then maybe one day using that as a basis for a W3C specification.
<jugglinmike> Matt_King: It's truly test-driven development!
<jugglinmike> Lola_Odelola: We wanted to understand the usage of MAY. The use of MAY might confuse some users
<jugglinmike> Lola_Odelola: The presence of MAY might cause folks to assume that other behaviors are not allowed
<jugglinmike> Matt_King: We might want to be more comprehensive in our assertions
<jugglinmike> Matt_King: if there are other responses to a specific attribute that are valid, then we might want to include them in the test
<jugglinmike> Matt_King: So I think actually, that hypotethical interpretation is valid
<jugglinmike> Matt_King: This places a greater onus on test writers to make sure that the universe of allowable things in included
<jugglinmike> Matt_King: But we could also write this specification in a way that doesn't exclude other possibilities
<jugglinmike> Hadi: I appreciate the use of the word "MUST", but I'm concerned about being too strict about specific wording
<jugglinmike> Matt_King: Agreed. That's why we use words like "convey". So even a "MUST" assertion allows for the kind of variations that you're referencing
<jugglinmike> Matt_King: MUST is the stuff like name, role, value, and state
<jugglinmike> Matt_King: Things like communicating change in state (rather than actually changing the state itself) would be "SHOULD"
<jugglinmike> Matt_King: It could be frustrating or maybe annoying if the AT doesn't do the things it SHOULD do, but the application would still be usable
<jugglinmike> Matt_King: And "MAY" assertions are essentially saying, "we don't care whether the AT does this"
<jugglinmike> Matt_King: using "MAY" increases our responsibility for capturing all the potential behaviors, and it also increases our responsibility for documentation (i.e. explaining what it means)
<jugglinmike> Matt_King: If there are any MAYs listed at all, there will be an implicit expectation that the assertions are exhaustive
<jugglinmike> James_Scholes: I think that puts an unreasonable burden on the test author--to account for anything a screen reader may do
<jugglinmike> James_Scholes: If we are tying assertions to any attribute, and some of them have the MAY priority, those are fringe usability benefits
<jugglinmike> James_Scholes: I don't think it should be the responsibility of the Test Author to enumerate every possible fringe usability benefits
<jugglinmike> Hadi: What do you folks think about mapping "MUST" "SHOULD" "MAY" to the concept of verbosity?
<jugglinmike> James_Scholes: I don't think that works for two reasons. First, not all screen readers have the same concept of verbosity (NVDA doesn't have such a concept at all). Secondly, some ATs have made decisions about verbosity which conflict with basic requirements
<jugglinmike> Alysssa: We test with the most verbose AT settings, right?
<jugglinmike> James_Scholes: That's right
<jugglinmike> s/Alysssa/Alyssa/
<jugglinmike> Matt_King: You're point is well-taken, James_Scholes. In that case, we would need language someplace that explains why some behaviors are recognized with MAY assertions and other behaviors are not mentioned, but they are nonetheless allowed
<jugglinmike> James_Scholes: I don't want to be in a position where we are claiming that we have thought of every possible innovation which an AT might make
<jugglinmike> Matt_King: We're going to need clear language around what "MAY" means. They are consensus suggestions as opposed to exclusive lists. We are going to have to write that before we publish anything to APG that includes a "MAY"
<jugglinmike> Zakim, end the meeting

[ccanash]

Closing as released to production