Improve readability of table that presents test results for each command

[mcking65]

Problem description

The design of the table that presents results for each command in a test creates several problems:

• The strings in the support column confuse readers. For example, see feedback from Vispero in JAWS Feedback: "Check a radio button in interaction mode" (Radio Group Example Using aria-activedescendant, Test 26, 10-04-2023) aria-at#966.
• The table is difficult to scan for failing assertions.
• It is difficult for screen reader users to efficiently navigate the information in the details column; there is too much information in a single cell.

This table is used in:

• Test runner to display submitted results.
• Candidate Review to show results for each test
• Report pages to show results for each test

Key assumptions

Note:

• These changes are intended to work with both V1 and the new V2 test format. This issue is the implementation that supports the V1 format.
• This solution description assumes that it is implemented before the following additional changes:
  • Revise reports to show assertion priorities as MUST/SHOULD/MAY instead of REQUIRED/OPTIONAL  #737
  • Count problematic (unexpected) behaviors as assertion failures #738
  • Support V2 test format when displaying reports #744
  • Update test builder to support V2 of test format aria-at#977

It assumes that both "no output" and "incorrect output" verdicts are presented as failed assertions as is described in #735.

Solution proposal

Instead of presenting results for all commands in a single table, present results for each command in a separate section containing the following elements.

1. Heading with:

   • Text: "COMMAND_NAME Results: X passed, Y failed
   • Example: Insert+UpArrow Results: 4 passed, 2 failed
   • if the command is a sequence, display it as "COMMAND1 then COMMAND2 then ... Results: X passed, Y failed"
   • Example: "Down Arrow then Down Arrow Results: 4 passed, 2 failed
     *Note: In current reports, the word "then" is preceded by a comma. Do not include a comma.
   • Note: Heading level should be determined by context. Using current IA, it would be level 3 in the test runner, candidate review, and report pages.

2. BlockQuote labeled "AT_NAME Response" and contains the output string.

3. Table captioned "COMMAND_NAME Results" and columns:

   • Priority: ASSERTION_PRIORITY
   • Assertion: ASSERTION_TEXT
   • Verdict: Passed | Failed
   • Rows are sorted by priority descending, then result (failures then passes), then assertion order. Assertion order refers to the order of assertion columns in the tests.csv file.

4. A paragraph element that:

   • If no unexpected behaviors were reported, contains text: "Unexpected behaviors: None"
   • If unexpected behaviors were reported, contains text "Unexpected behaviors:", and it is followed by an unordered list containing the behaviors.

Additional integration requirements

In the test runner and candidate review experiences, the results tables are preceded by a level 2 heading with text:

Test Result: PASS
or
Test Result: FAIL

Change this heading text to:

Test Results (x passed, y failed)

Where x and y represent the total number of passing and failing assertions across all commands in the test.

Currently, on report pages, the test results table is preceded by a level 2 heading with text:

Details for test:TEST_NAME

Example:

Details for test:Trigger an alert in reading mode

Change the text of this heading to:

TEST_NAME (X passed, y failed)

Example using current JAWS/Chrome results:

Trigger an alert in reading mode (2 passed, 2 failed)

Mockup

Here is a zip containing an HTML mockup with data from the radio group report.
TestResultsTableMockupForIssue733.zip

[mcking65]

@jscholes

We'll discuss in our Tuesday meeting. If possible, please preview before we meet.

[ccanash]

Closing as released to production