Support for command sequences in tests

[jscholes]

Currently, the test format allows for multiple commands, but it is implied that these should be executed in isolation. For example, pressing F to find a checkbox, returning to a known point on the page and then pressing X to find the same control.

this isn't sufficient in some cases, and multiple commands are required in a sequence instead. Three categories that spring to mind are:

1. AT virtual buffer structure creating the need for multiple consecutive executions of the same command. For example, JAWS requires multiple presses of Down Arrow to read the information about a table container as well as the text and coordinates of the first cell. The same applies to VO+Left/Right Arrow when moving into certain components.
2. AT behaviour creating the need for one key to be pressed followed by another. To reuse the table example from point #1, a JAWS tester will need to press T followed by Down Arrow to locate the table and move into the first cell.
3. Multiple commands needed in a certain time period/window. For example, type-ahead using multiple characters in a listbox.

Note that for points #1 and #2, the intermediate output is needed for testers to be able to make a verdict about the assertions. For example after pressing T to find a table, testers should listen to the information about the table before pressing the next command to hear information about the first cell. But for point #3, we generally only care about the speech output when the final destination is reached (e.g. the listbox option matching all typed characters).

Currently, this can be hacked via an entry in keys.mjs, e.g.:

export const T_THEN_DOWN = "T followed by Down Arrow";

This has multiple downsides, including:

• It's non-obvious for new test writers. Even if we were to document it, it would still be a hack, and may introduce inconsistencies if different test authors used different wording.
• It makes the keys file, which is already in need of some organisation, crowded.
• It's a blunt tool which doesn't really allow distinction between the three use cases outlined above.
• It will frustrate any future plans to change the wording for all command sequences, localise the testing UI, etc.

[mfairchild365]

This could be relevant to automation. I bet that using an array of commands would be easier to process from an automation perspective.

[jscholes]

Points of interest from the February 25, 2021 community group meeting:

• We don't want so-called "open-ended" sequences, where a human or automated tester is instructed to press a key multiple times until a condition is met. Instead, all testing tasks should encode an explicit number of commands, even if it is the same command a set number of times.
• To that end, commands.csv needs to accept 1 or more commands in each individual column, as opposed to just one which is the current behaviour.
• When a task includes multiple commands in a sequence, assertions should be evaluated based on aggregate output from all of those commands.
• Too much cognitive overhead to ask a human tester to enter output from separate commands into multiple textboxes on the results form. But the textbox is now multiline, so testers can report output of multiple commands there. When a test is automated though, storing screen reader output mapped to the command that triggered it will be easier.
• Some commands won't contribute to the passing or failing of assertions, e.g. because they're a part of setup steps and so we don't really care about the output. We don't currently have support for programmatically encoding setup commands; need example cases of where a setup script isn't appropriate on its own to determine if this is needed or not.
• Ideas:
  • Have two fields in the data model, one for screen reader output determined by human testers and one for the same output confirmed via automation. The second could be stored as an array/map based on the commands that triggered each piece of output.
  • Create screen reader extensions (e.g. an NVDA add-on) which helps human testers capture output rather than relying on them typing/pasting it manually. This add-on could create the mapping between commands and speech sequences, and maybe even use separators in the multiline textbox to programmatically indicate the boundaries between commands for storage.
  • Come up with proposal for the above capture output system to be included in ATs by default.

Proposed technical solution:

Allow for some sort of array syntax in commands.csv, like square brackets or braces, to wrap multiple commands in a single column. Note that if this is comma-separated (which it most likely should be), the array will need quotes anyway. So maybe we could just get rid of the idea of brackets/braces, and have the scripts determine whether a command contains a comma? If it does, it's a sequence. Example:

1,navigate forwards to grid,reading,JAWS,"T,DOWN","DOWN,DOWN",,,,

The above has two command sequences:

• T followed by Down Arrow; and
• Down Arrow, twice.

With brackets, it would look like:

1,navigate forwards to grid,reading,JAWS,"[T,DOWN]","[DOWN,DOWN]",,,,

Given that both examples need the quotes, the only difference is that the presence of an array/sequence is made a tad more explicit in the second one via the inclusion of brackets. Given that these files should be read by a machine most of the time, I'm not sure I'd even call this an advantage.

[jscholes]

I should note, we didn't get to discussing one of the points raised by this issue on the Feb 25 meeting:

Multiple commands needed in a certain time period/window. For example, type-ahead using multiple characters in a listbox.

[mfairchild365]

I agree that we don't need the brackets.

[jscholes]

Some notes from the March 11, 2021 CG meeting: we don't need to test type-ahead-style functionality at this time. Testing a single key, e.g. to jump to an option in a listbox, is sufficient. This will be far easier to implement within the framework of automated testing than it will be to instruct human testers to press a series of keys in a very short window of time.

[zcorpan]

PR: #438

[jscholes]

Reopening based on the context in #537 (comment) and #537 (comment); it seems that for some reason the support for command sequences is not actually present despite PR #438. CC @sinabahram / @s3ththompson / @richnoah, and also @zcorpan who wrote the PR.