Create tests for APG design pattern example: Tri-State Checkbox

[jscholes]

Applies To

• APG design pattern: Checkbox
• Specific example: Tri-State Checkbox

Testing Notes

This issue thread will house all relevant documentation relating to the development of a test plan for the "Tri-State Checkbox" APG example, which conforms to the "Checkbox" design pattern. All follow-ups (including the test plan itself) will be added as new comments to facilitate notifications, and this initial comment will be updated with relevant details and URLs as needed for convenience.

Useful Links

• Previewable Tests Index
• Previewable Tests Review Page

Additional References

• Checkbox (role)
• aria-checked (state)

[jscholes]

On the linked sample, aria-controls is used on the element with role="checkbox", despite not being mentioned within the design pattern. I've filed an issue for clarification about this: w3c/aria-practices#1599

My belief is that no assertions relating to aria-controls can be included in a test plan as of the time of filing, because there is no meaningful understanding of how to surface the presence of multiple IDREFs in a user-friendly way.

[IsaDC]

Below is the test plan for the Tri-State Checkbox example. Some notes/context to be aware of:

• All navigation-related tests have been split based on direction, following discussion with @mcking65. For example, Tab and Shift+Tab are tested separately rather than together, to avoid testers having to infer how to move their focus/virtual cursor after/outside of a particular element or container.
• JAWS tests with Smart Navigation enabled have been omitted and won't be included going forward.
• A setup script has been specified for all forward-navigation tests, to place keyboard focus on a link before the tri-state checkbox. This avoids testers needing to pay attention to how many times they've pressed keys like Tab, potentially resulting in data degradation if testers are distracted.
• These tests are intended to compliment those for the two-state checkbox pattern example while avoiding unnecessary repetition. For instance, we guide users in testing the state transitions between partially checked and checked, and not checked to partially checked, as these are both unique to tri-state checkboxes. But we don't test for not checked to checked and vice versa as it would be redundant.
• We also don't test the state transition between partially checked and not checked, as this is not possible with one keystroke and may violate user expectations. We could advise testers to uncheck all of the individual two-state checkboxes and then navigate back to the tri-state control, but we would only end up recreating the existing "navigate to an unchecked checkbox" test from the two-state test plan.
• The tests relating to grouping have been included, with the following caveats:
  • The tri-state checkbox example uses a <fieldset> and <legend> rather than role="group". As such, we are implicitly testing HTML semantics in all of the group-related tests, not WAI-ARIA. The two-state checkbox example uses role="group" and aria-labelledby, and this example should probably be updated to follow the same mark-up. Alternatively, the group-related tests can be removed from this test plan.
  • Navigating forwards out of a group and navigating backwards into a group are not tested, as they would require the tester to use up to five separate keystrokes during a test to bypass the two-state checkboxes.
  • Up and Down Arrow are not tested for exiting or entering a group, nor Control+Option+Left Arrow, Control+Option+Right Arrow, Control+Option+Command+J and Control+Option+Command+Shift+J on the Mac, as the use of a <legend> will impose an additional element between the start of the group and tri-state checkbox when navigating with screen reader cursors.

Each test title is marked up as a level 4 heading to facilitate screen reader navigation.

Navigate forwards to a partially checked checkbox in reading mode

• Applies to: JAWS, NVDA
• Mode: reading
• Instructions: Navigate to the first checkbox. Note: it should be in the partially checked state. You may need to press certain commands (such as Down Arrow) multiple times to hear all information relating to the checkbox.
• Commands:
  • X
  • F
  • Tab
  • Down Arrow
• Setup script description: Sets focus on a link before the first checkbox

Assertions

Assertion	Priority	Reference
Role 'checkbox' is conveyed	required	role="checkbox"
Name 'All condiments' is conveyed	required	Visible text content contained within the element with role="checkbox"
State of the checkbox (partially checked) is conveyed	required	aria-checked="mixed"

Navigate backwards to a partially checked checkbox in reading mode

• Applies to: JAWS, NVDA
• Mode: reading
• Instructions: Navigate to the first checkbox. Note: it should be in the partially checked state.
• Commands:
  • Shift+X
  • Shift+F
  • Shift+Tab
  • Up Arrow
• Setup script description: Sets focus on a link after the first checkbox

Assertions

Assertion	Priority	Reference
Role 'checkbox' is conveyed	required	role="checkbox"
Name 'All condiments' is conveyed	required	Visible text content contained within the element with role="checkbox"
State of the checkbox (partially checked) is conveyed	required	aria-checked="mixed"

Navigate forwards to a partially checked checkbox in interaction mode

• Applies to: JAWS, NVDA
• Mode: interaction
• Instructions: Navigate to the first checkbox. Note: it should be in the partially checked state.
• Commands:
  • Tab
• Setup script description: Sets focus on a link before the first checkbox

Assertions

Assertion	Priority	Reference
Role 'checkbox' is conveyed	required	role="checkbox"
Name 'All condiments' is conveyed	required	Visible text content contained within the element with role="checkbox"
State of the checkbox (partially checked) is conveyed	required	aria-checked="mixed"

Navigate backwards to a partially checked checkbox in interaction mode

• Applies to: JAWS, NVDA
• Mode: interaction
• Instructions: Navigate to the first checkbox. Note: it should be in the partially checked state.
• Commands:
  • Shift+Tab
• Setup script description: Sets focus on a link after the first checkbox

Assertions

Assertion	Priority	Reference
Role 'checkbox' is conveyed	required	role="checkbox"
Name 'All condiments' is conveyed	required	Visible text content contained within the element with role="checkbox"
State of the checkbox (partially checked) is conveyed	required	aria-checked="mixed"

Navigate forwards to a partially checked checkbox

• Applies to: MacOS VoiceOver
• Mode: interaction
• Instructions: Navigate to the first checkbox. Note: it should be in the partially checked state. You may need to press certain commands (such as Control+Option+Right) multiple times to hear all information relating to the checkbox.
• Commands:
  • Tab
  • Control+Option+Right Arrow
  • Control+Option+Command+J
• Setup script description: Sets focus on a link before the first checkbox

Assertions

Assertion	Priority	Reference
Role 'checkbox' is conveyed	required	role="checkbox"
Name 'All condiments' is conveyed	required	Visible text content contained within the element with role="checkbox"
State of the checkbox (partially checked) is conveyed	required	aria-checked="mixed"

Navigate backwards to a partially checked checkbox

• Applies to: MacOS VoiceOver
• Mode: interaction
• Instructions: Navigate to the first checkbox. Note: it should be in the partially checked state.
• Commands:
  • Shift+Tab
  • Control+Option+Left Arrow
  • Shift+Control+Option+Command+J
• Setup script description: Sets focus on a link after the first checkbox

Assertions

Assertion	Priority	Reference
Role 'checkbox' is conveyed	required	role="checkbox"
Name 'All condiments' is conveyed	required	Visible text content contained within the element with role="checkbox"
State of the checkbox (partially checked) is conveyed	required	aria-checked="mixed"

Operate a partially checked checkbox in reading mode

• Applies to: JAWS, NVDA
• Mode: reading
• Instructions: Check the first checkbox.
• Commands:
  • Space
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Change in state (to checked) is conveyed	required	aria-checked="true"

Operate a partially checked checkbox in interaction mode

• Applies to: JAWS, NVDA
• Mode: interaction
• Instructions: Check the first checkbox.
• Commands:
  • Space
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Change in state (to checked) is conveyed	required	aria-checked="true"

Operate a partially checked checkbox

• Applies to: MacOS VoiceOver
• Mode: interaction
• Instructions: Check the first checkbox.
• Commands:
  • Control+Option+Space
  • Space
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Change in state (to checked) is conveyed	required	aria-checked="true"

Operate an unchecked checkbox in reading mode

• Applies to: JAWS, NVDA
• Mode: reading
• Instructions: Partially check the first checkbox.
• Commands:
  • Space
• Setup script description: sets focus on the first checkbox and sets its state to unchecked

Assertions

Assertion	Priority	Reference
Change in state (to partially checked) is conveyed	required	aria-checked="mixed"

Operate an unchecked checkbox in interaction mode

• Applies to: JAWS, NVDA
• Mode: interaction
• Instructions: Partially check the first checkbox.
• Commands:
  • Space
• Setup script description: sets focus on the first checkbox and sets its state to unchecked

Assertions

Assertion	Priority	Reference
Change in state (to partially checked) is conveyed	required	aria-checked="mixed"

Operate an unchecked checkbox

• Applies to: MacOS VoiceOver
• Mode: interaction
• Instructions: Partially check the first checkbox.
• Commands:
  • Control+Option+Space
  • Space
• Setup script description: sets focus on the first checkbox and sets its state to unchecked

Assertions

Assertion	Priority	Reference
Change in state (to partially checked) is conveyed	required	aria-checked="mixed"

Read a partially checked checkbox in reading mode

• Applies to: JAWS, NVDA
• Mode: reading
• Instructions: When the reading cursor is on the first checkbox, read the first checkbox. Note: it should be in the partially checked state.
• Commands:
  • Insert+Tab
  • Insert+Up Arrow
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Role 'checkbox' is conveyed	required	role="checkbox"`
Name 'All condiments' is conveyed	required	Visible text content contained within the element with role="checkbox"
State of the checkbox (partially checked) is conveyed	required	aria-checked="mixed"

Read a partially checked checkbox in interaction mode

• Applies to: JAWS, NVDA
• Mode: interaction
• Instructions: When the focus is on the first checkbox, read the first checkbox. Note: it should be in the partially checked state.
• Commands:
  • Insert+Tab
  • Insert+Up Arrow
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Role 'checkbox' is conveyed	required	role="checkbox"`
Name 'All condiments' is conveyed	required	Visible text content contained within the element with role="checkbox"
State of the checkbox (partially checked) is conveyed	required	aria-checked="mixed"

Read a partially checked checkbox

• Applies to: MacOS VoiceOver
• Mode: interaction
• Instructions: When the focus is on the first checkbox, read the first checkbox. Note: it should be in the partially checked state.
• Commands:
  • Control+Option+F3
  • Control+Option+F4
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Role 'checkbox' is conveyed	required	role="checkbox"`
Name 'All condiments' is conveyed	required	Visible text content contained within the element with role="checkbox"
State of the checkbox (partially checked) is conveyed	required	aria-checked="mixed"

Read grouping information of a grouped, partially checked checkbox in reading mode

• Applies to: JAWS, NVDA
• Mode: Reading
• Instructions: When reading cursor is on the first checkbox, read its grouping information
• Commands:
  • Insert+Tab
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Role 'group' is conveyed	optional	<fieldset>
Group name 'Sandwich Condiments' is conveyed	required	Visible text content contained within a <legend>, as a child of a <fieldset>

Read grouping information of a grouped, partially checked checkbox in interaction mode

• Applies to: JAWS, NVDA
• Mode: interaction
• Instructions: When reading cursor is on the first checkbox, read its grouping information
• Commands:
  • Insert+Tab
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Role 'group' is conveyed	optional	<fieldset>
Group name 'Sandwich Condiments' is conveyed	required	Visible text content contained within a <legend>, as a child of a <fieldset>

Read grouping information of a grouped, partially checked checkbox

• Applies to: MacOS VoiceOver
• Mode: interaction
• Instructions: When reading cursor is on the first checkbox, read its grouping information
• Commands:
  • Control+Option+F3
  • Control+Option+F4
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Role 'group' is conveyed	optional	<fieldset>
Group name 'Sandwich Condiments' is conveyed	required	Visible text content contained within a <legend>, as a child of a <fieldset>

Navigate forwards into a checkbox group in reading mode

• Applies to: JAWS, NVDA
• Mode: reading
• Instructions: Navigate forwards into the checkbox group.
• Commands:
  • X
  • F
  • Tab
• Setup script description: Sets focus on a link before the first checkbox

Assertions

Assertion	Priority	Reference
Role 'group' is conveyed	required	<fieldset>
Group name 'Sandwich Condiments' is conveyed	required	Visible text content contained within a <legend>, as a child of a <fieldset>
The group boundary is conveyed	required	<fieldset>

Navigate backwards out of a checkbox group in reading mode

• Applies to: JAWS, NVDA
• Mode: reading
• Instructions: Navigate backwards out of the checkbox group.
• Commands:
  • Shift+Tab
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Role 'group' is conveyed	optional	<fieldset>
Group name 'Sandwich Condiments' is conveyed	optional	Visible text content contained within a <legend>, as a child of a <fieldset>
The group boundary is conveyed	optional	<fieldset>

Navigate forwards into a checkbox group in interaction mode

• Applies to: JAWS, NVDA
• Mode: interaction
• Instructions: Navigate forwards into the checkbox group.
• Commands:
  • Tab
• Setup script description: Sets focus on a link before the first checkbox

Assertions

Assertion	Priority	Reference
Role 'group' is conveyed	required	<fieldset>
Group name 'Sandwich Condiments' is conveyed	required	Visible text content contained within a <legend>, as a child of a <fieldset>
The group boundary is conveyed	required	<fieldset>

Navigate backwards out of a checkbox group in interaction mode

• Applies to: JAWS, NVDA
• Mode: interaction
• Instructions: Navigate backwards out of the checkbox group.
• Commands:
  • Shift+Tab
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Role 'group' is conveyed	optional	<fieldset>
Group name 'Sandwich Condiments' is conveyed	optional	Visible text content contained within a <legend>, as a child of a <fieldset>
The group boundary is conveyed	optional	<fieldset>

Navigate forwards into a checkbox group

• Applies to: MacOS VoiceOver
• Mode: interaction
• Instructions: Navigate forwards into the checkbox group. Note: You may need to press certain commands (such as Control+Option+Command+J) multiple times to hear all information relating to the checkbox.
• Commands:
  • Tab
  • Control+Option+Command+J
• Setup script description: Sets focus on a link before the first checkbox

Assertions

Assertion	Priority	Reference
Role 'group' is conveyed	required	<fieldset>
Group name 'Sandwich Condiments' is conveyed	required	Visible text content contained within a <legend>, as a child of a <fieldset>
The group boundary is conveyed	required	<fieldset>

Navigate backwards out of a checkbox group

• Applies to: MacOS VoiceOver
• Mode: interaction
• Instructions: Navigate backwards out of the checkbox group.
• Commands:
  • Shift+Tab
  • Shift+Control+Option+Command+J
• Setup script description: sets focus on the first checkbox

Assertions

Assertion	Priority	Reference
Role 'group' is conveyed	optional	<fieldset>
Group name 'Sandwich Condiments' is conveyed	optional	Visible text content contained within a <legend>, as a child of a <fieldset>
The group boundary is conveyed	optional	<fieldset>

[jscholes]

Additional note: we agreed with @mcking65 that JAWS tests with Smart Navigation enabled will be omitted going forward.

[jscholes]

These tests are now ready for review. For convenience, I've added links to the previewable index and review pages in the initial comment:

• Previewable Tests Index
• Previewable Tests Review Page