Update test queue to support generating reports that require a specific AT or browser version

[mcking65]

ARIA-AT goal

One goal of the ARIA-AT working mode specific to test plans in the recommended phase is to keep their interop reports current. That means, for each AT/Browser combo that has a report, generate a new report when a new version of an AT or browser is released. In addition to keeping data current, over time, this will also create data that can be used to publish trend charts.

The CG has not yet defined criteria for which releases of AT or browsers would updates. That criteria could change over time.

As long as reporting involves a substantial amount of manual work, the community group may choose to be fairly selective about the releases for which new reports are required. For example, we could require new reports only for major AT releases and not for new browser releases. Any recent browser version could be considered sufficient in that scenario.

Test queue design update objective

Enable the test queue to include test plan runs that require a specific version of an AT or browser. If a test plan run does not require a specific version of either AT or browser, clearly indicate that any recent version is sufficient.

Design discussion

Should we fit version requirements into the current tables? Or, should we use this issue to drive a redesign of the information architecture of the queue?

Expedient option A: Add a table column "Version Requirements". Show the AT /browser name and versions required.

Examples:

• JAWS Any Recent/ChromeAny recent
• JAWS 2023.2303.144 /Chrome Any Recent
• JAWS 2023.2303.144.400/Chrome 116.0.5845.188

Note that when specific versions are required, opening the test plan should require the user to confirm their versions match the requirements.

Option B: Consider restructuring the information on the test queue page to better align with how the CG has been working, e.g., a heading for each test plan and a table with columns for AT, Browser, assignees, status, and actions. The version information could be specified in the AT and Browser columns.

[css-meeting-bot]

The ARIA-AT Community Group just discussed Issue 791: Generating reports for specific AT/Browser versions.

The full IRC log of that discussion

<jugglinmike> Topic: Issue 791: Generating reports for specific AT/Browser versions
<jugglinmike> github: https://github.com//issues/791
<jugglinmike> Matt_King: Right now, everything we've been dealing with is reviewing drafts and getting them to candidate
<jugglinmike> Matt_King: We haven't cared too much about browser and AT versions; we've just recorded what Testers have been using
<jugglinmike> Matt_King: That's fine until we reach the "Recommended" phase of a report
<jugglinmike> Matt_King: At that phase, we're reporting to the world how this software works
<jugglinmike> Matt_King: The expectation we've had from the beginning is that once we have a "Recommended" test plan, we're going to keep that data current indefinitely
<jugglinmike> Matt_King: At that point, it becomes important for us to re-run the reports when each AT releases a new version and (perhaps to a slightly less critical extent) when each browser releases a new version
<jugglinmike> Matt_King: So, this can be a little bit challenging, and there are a lot of decisions to make in this space
<jugglinmike> Matt_King: Let's talk about screen readers, first
<jugglinmike> Matt_King: Right now, when we add a plan to the queue, we don't specify a screen reader version on purpose
<jugglinmike> Matt_King: (We had this ability three years ago, but we removed it because it was causing problems for us)
<jugglinmike> Matt_King: We still want to say "any recent version is fine" for draft and candidate
<jugglinmike> Matt_King: But when we reach "recommended", we need to enforce specific versions
<jugglinmike> Matt_King: One problem is that we don't have a way in the test queue that a specific version is required (that seems pretty easy to solve)
<jugglinmike> Matt_King: It also means requiring Testers to confirm that they're using the appropriate version
<jugglinmike> Matt_King: Thirdly, the Test Admin needs to choose a specific version make new Test Plans
<jugglinmike> James_Scholes: that all seems right to me
<jugglinmike> Matt_King: Okay, moving on to browsers.
<jugglinmike> Matt_King: All of this gets so much harder because (1) there are a lot more browsers, and (2) because we will need to maintain a list of browser versions, and (3) a lot of people don't have control over the version of the browser which is installed on their system
<jugglinmike> James_Scholes: for regular users who are not using corporate machines, it is quite difficult to install a specific version of a browser (first because they aren't always published--as in the case of Chrome). Secondly, even if you manage to do that, it's probably going to update itself, anyway
<jugglinmike> James_Scholes: I think that's separate from the "enterprise problem" where Testers are working in a corporate setting where there is strict external control over the applications installed on their machine
<jugglinmike> James_Scholes: I think it's really difficult to ask people to use a specific browser build. Even requiring just a major version is tricky
<jugglinmike> Matt_King: But we could ask machines to do this
<jugglinmike> Matt_King: I'm thinking about a decision that we could make. For the time being--until we have full automation across the board--we only ever require specific AT versions...
<jugglinmike> Matt_King: ...that we TRACK the browser versions (just like we do now), but we don't require it
<jugglinmike> James_Scholes: If we say that, "oh, we don't want someone testing with a very outdated version of Chrome", I think that may be a good middle ground
<jugglinmike> James_Scholes: Because if we go super-specific on browser version requirements, we'd probably also have to requirements on operating system versions
<jugglinmike> jugglinmike: We might want to track version release dates as well, because "four major versions old" means something very different for Chrome and for Safari
<jugglinmike> jugglinmike: The kind of warning we're discussing (telling Testers that they're browser is suspiciously old) will be hard to express for a single "version delta" across all browsers. Expressing it in terms of release date will be more meaningful
<jugglinmike> Matt_King: I agree!
<jugglinmike> Matt_King: So for the foreseeable future, we will not require updating the data for Recommended Reports in response to the release of new browser versions. We will only require updating in response to new versions of ATs...
<jugglinmike> Matt_King: And in that case, we will accept "any recent" version of the browsers under test
<jugglinmike> James_Scholes: sounds good
<jugglinmike> IsaDC: sounds good
<jugglinmike> Lola_Odelola: sounds good
<jugglinmike> Matt_King: Okay, that's good. This helps a lot. I'll update my work to consistently use the terminology "any recent browser"
<jugglinmike> Matt_King: Thanks, everyone!
<jugglinmike> Zakim, end the meeting

[howard-e]

Discussions outside this issue and also described in #792 note that specifying the browser version will not be required for the short term, and we're going ahead with option A for the design consideration.

[mcking65]

Since encountering the conditions that led to w3c/aria-at#1047, and recalling comments that @jscholes has made in the past about clearly defining "recent", I want to open discussion about changing "any recent" to "VX or Later".

In other words, when adding a run to the queue in draft or candidate review, the admin would be able to choose between specifying either a minimum version or a required version for an AT. Then in the test queue table, for each run, the AT and version cell in the table would say something like "JAWS 2024.2310.70.400 or Later" or "JAWS 2024.2310.70.400".

This would eliminate ambiguity about version requirements.

If we go this route, it would be the responsibility of the community group to maintain a list of what it considers to be the most appropriate minimum versions to use during draft review. It would be the responsibility of the admin to specify those minimums when adding a run to the queue.

One way of doing this that would enable consistency across test runs with minimal administrative effort would be to enhance the "Manage Assistive Technology Versions" feature on the data management page so that we can specify which version is the current "Minimum Required for Draft Review". When a new version of an AT is released and the admin is adding it to the system, that would also be the time that an admin may consider changing what the current minimum is for future draft review.

By having the system keep track of which version of an AT is the current required minimum version for review, it would simplify the process of adding a run to the test queue. Admins would not have to remember what the community group recommends as the currently required minimum. When adding a run to the queue, an admin would be able to choose one of two version requirements: option 1: "Minimum Required Version or Later" or option 2: a specific version.

I'd like to discuss this with @jscholes and @IsaDC before we finalize #952. James and Isabel, I will have this on our agenda for Tuesday morning.

[mcking65]

I think we should consider going with design option B instead of option A for two reasons:

1. It is getting increasingly difficult to manage the test queue because the CG work is now primarily organized by test plan, not by AT/browser combination. During meetings, we have to look across multiple tables to get the information we need to make decisions, and we are having to do too much back and forth between data management and test queue.
2. I am hesitant to invest any more design and engineering resource in the current test queue page because it is already apparent that the current test queue organization is not aligned with the working mode and it is pretty obvious that organizing by test plan is more appropriate.

I'd like to discuss the following proposal for reorganizing the test queue with @jscholes and @IsaDC before we finalize #952. It doesn't seem like this proposal would be much of a lift because it is all the same data using all the same UI components; they are simply arranged differently.

Test Queue Option B Proposal

The test queue page would be a series of tables like it is now, but instead of having tables that contain data for all runs for a given AT/browser combination, each table would contain data for all the runs for a specific test plan version.

Each test plan version that has one or more runs in the test queue would have a collapsible section where:

1. The section heading is an H2 with the test plan name, version, and phase, e.g. "Toggle Button V24.03.12 in Draft Review"
2. The section heading is a toggle that expands/collapses the section like the H2 elements for each version on a Test Plan Versions page.
3. The section content includes:
   1. A link to the test plan, e.g., View V24.03.12 of the Toggle Button Test Plan
   2. A button that opens the report status dialog. This is exactly the same button that appears in the corresponding cell in the data management table.
   3. The order of sections would be alphabetical by test plan name and then by descending version. So, if there are two versions of the same test plan with runs in the queue, one in draft review and one in candidate, the newest test plan version section would be first.
   4. The test plan runs table described below. The table has an accessible name equal to the section heading, e.g., has aria-labelledby pointing to the H2.

The test plan runs table for a test plan version has the following columns sorted by AT name ascending:

1. AT: AT name and version requirement. Cells in this column are row headers, i.e., are th elements.
2. Browser: browser name and version requirement
3. Testers: contains the same content as the cells in the testers column of the current test queue tables.
4. Status: contains the same content as the cells in the status column of the current test queue tables with one exception noted below.
5. Actions: contains the same content as the cells in the actions column of the current test queue tables.

Note about status column: The current status includes strings like "Draft" or "Has Conflicts". Replace that text portion of the status with the string "X% complete by Y testers with Z conflicts" where Percent complete = (NUMBER_COMPLETED_TESTS_BY_ALL_TESTERS / (NUMBER_ASSIGNED_TESTERS * NUMBER_TESTS_IN_PLAN)) * 100.

Here's an example: test queue option B proposal.zip

[IsaDC]

@mcking65 Agree with these changes.

[isaacdurazo]

Here's a Mockup for Option B Proposal.

@mcking65 This mockup looks exactly like your html example. I only made one additon, which is a progress bar above "X% complete by Y testers with Z conflicts" in the status column. Cc @ccanash