Design Missing Test Plan Status Management Functions

[mcking65]

Related to #518.

UPDATED JUNE 14

June 14 revision summary

• Improved column titles.
• Changed plan version strings to "Vyy.mm.dd format so they are shorter and to prevent them from being confused with status dates.
• Minor revision to content of the "Overall Status" column, which required a significant revision to business logic for calculating status.
• Shortened and clarified label of button that opens report status dialog. Moved description of the button labeling logic to the section that describes the dialog.
• Shortened label of button that opens candidate target dialog.
• Reduce content in most cells of the phase columns.
• Added a requirement related to confirmation prompts when changing the phase of a version.
• Bumped priority of the "Test Plan Versions" page from P2 to P1. This page will be documented in a separate issue.
• Reduced priority of filters from P1 to P2.

Test Plan Status Table Proposal Overview

This issue describes a proposal for the test plan status summary table, including:

• The columns to include
• The content of the columns
• Default sort order
• Filters

It also introduces a new concept of "required reports" to help manage progress of a test plan through the working mode process.

Development priorities

Each feature of the table is marked with a priority of P0, P1, or P2.
The assumption is that we will develop a short list of features in each sprint, with the first several sprints each delivering a set of P0 requirements and later sprints focused on P1 and P2 requirements.

Overview of Columns for Test Plan Status Summary Table

1. Test Plan
2. Covered AT
3. Overall Status
4. R&D Version
5. Draft Review
6. Candidate Review
7. Recommended Version

General Description of Phase Columns

Note: As described in w3c/aria-at#950, a test plan may have up to one version in each phase of the working mode. For example, there may be a recommended version and a new version that is ready for draft review at the same time.

The Hhigh-level objectives of the columns for the R&D, Draft, Candidate, and Recommended phases is to enable users to monitor progress of a test plan through the working mode and enable admins to manage the process.

While the content of each column is different, the common objectives are to show:

• Whether the test plan has a version in that phase of the working mode.
• If there is not a version in the phase, shows if and when a version last completed that phase.
• If there is a version in the phase:
  • Show the version identifier and provide access to a view of that version of the plan
  • Show status of relevant working mode requirements and provide access to relevant reports
  • Provide admins with relevant actions

As described in the detailed descriptions of column content that follows, each phase column may thus have:

• If the plan does not have a version in the phase, a simple status text string
• If the plan does have a version in the phase, a link and up to 3 buttons or strings.

Test Plan Column Details

• P0 requirement: Show the name of the test plan
• P1 Requirement: The name is a Link to the "Test Plan Versions" page, which will provide more details about the plan and all its versions. A description of this page will be provided in a separate issue.
• P2 requirement: table is sortable by the values in this column

Covered AT Column Details

• P0 requirement: As a near term stopgap, List covered AT. Works for now since we only have 3.
• P1 requirement: Later, we will define categories of AT and show the covered category, how many AT are covered within the category, and a button that popsup the list of them. For example, there could be a button named "3 Desktop Screen Readers" and activating it would open a dialog listing the screen readers.
• P2 requirement: table is sortable and filterable by the values in this column

Overall Status Column Details

P0 Requirement: Show the most advanced phase this plan has completed using the following logic:

• If there is a recommended version:
  • Show string "Recommended Since DATE" where DATE is the date on which a version of this plan was first recommended. Note that multiple versions may have progressed to recommended, so this is the date on which the earliest version progressed to recommended.
  • If there are also versions currently active in any of the candidate, draft, or R&D complete phases, append string "+N New Versions in Progress" where N is the number of phases where a version is active. If N=1, use singular "version".
• else If there is a candidate version:
  • Show string "Candidate Review Started DATE" where DATE is the date on which the candidate review phase began. Note that multiple versions may have progressed to candidate, so this is the date on which the earliest version progressed to candidate.
  • If there are also versions currently active in either the draft or R&D complete phases, append string "+N New Versions in Progress" where N is the number of phases where a version is active. If N=1, use singular "version".
• else If there is a draft version:
  • Show string "Draft Review Started DATE" where DATE is the date on which the draft review phase began. Note that multiple versions may have progressed to draft, so this is the date on which the earliest version progressed to draft.
  • If there is also a version currently active in the draft phase, append string "+1 New Version in Progress".
• else if R&D is complete, show string "R&D Complete on DATE" where DATE is the date from the version identifier of the version that is currently ready for draft review.

Notes:

• dates are sufficient; no need for time stamp.
• this column is used in the default sort order described below. The sort function of this column would return the table to its default state.

R&D Version Column Details

• P0 requirement: If the latest version of the plan is in the draft, candidate, or recommended phase, show string "N/A".
• P0 requirement: else If the latest version has completed R&D and has not yet been advanced to draft review, show:
  • Link with text "VERSION_STRING" that targets the single-page view of the plan. Example: V22.12.08
  • If current user is an admin, show "Advance to Draft" button. Note that:
    • If there is an earlier version that is draft and that version has some test plan runs in the test queue, this button will run the process for updating existing reports and preserving data for tests that have not changed.
    • if there is an earlier version in the draft phase, this button will sunset that version. This will also sunset any reports completed using that version.

Draft Review Column Details

• P0 requirement: If a version of the plan is not in the draft phase and there are no versions in later phases, show string "Not Started"
• P0 requirement: If a version of the plan is not in the draft phase and there is a version in at least one of candidate or recommended phases, show string "Review of VERSION_STRING completed DATE" where:
  • VERSION_STRING identifies the version that most recently completed draft review.
  • DATE is the date that the version identified by VERSION_STRING completed draft review and advanced to candidate review.
  • Example: "Review of V22.12.08 completed Jan 30, 2023"
• If there is a version currently in the draft phase, show:
  • P0 Requirement: Link with text "VERSION_STRING" that targets the single-page view of the plan. Example: V22.12.08
  • P1 requirement: Show a button that opens the test plan report status dialog. The button label and dialog are described below.
  • P0 requirement: If required reports are complete and user is an admin, show "Advance to Candidate" button. Note that:
    • If there is an earlier version that is candidate and that version has some test plan runs in the test queue, this button will run the process for updating existing reports and preserving data for tests that have not changed.
    • if there is an earlier version in the candidate phase, this button will sunset that version. This will also sunset any reports completed using that version.

Candidate Review Column Details

• P0 requirement: If a version of the plan is not in the candidate phase and there has not yet been a recommended version, show string "Not Started"
• P0 requirement: If a version of the plan is not in the candidate phase and there is a recommended version, show string "Review of VERSION_STRING completed DATE" where:
  • VERSION_STRING identifies the version that most recently completed candidate review.
  • DATE is the date that the version identified by VERSION_STRING completed draft review and advanced to candidate review.
  • Example: "Review of V22.12.08 completed Jan 30, 2023"
• If there is a version currently in the candidate phase, show:
  • P0 Requirement: Link with text "VERSION_STRING" that targets the single-page view of the plan. Example: V22.12.08
  • P0 Requirement: Show string "N Open Review Issues" and if N>=2, append " from N AT" Examples: "3 Open Review Issues from 2 AT" or "0 Open Review Issues"
  • P1 requirement: Show timeline string "Target N days away|past", Examples: "Target 90 days away" or "Target 10 days past"
  • P1 Requirement: If user is admin, the timeline string is a button that opens the change target date dialog.
  • P0 Requirement: Show button "Advance to Recommended" when the following conditions are met:
    • If there has not yet been a recommended version and open issues = 0 and days in review > 120 and user is admin, show the button.
    • If there is already a recommended version and open review issues = 0 and user is admin, show the button.
    • If there is an earlier version that is recommended and that version has some test plan runs in the test queue, this button will run the process for updating existing reports and preserving data for tests that have not changed.
    • if there is an earlier version in the recommended phase, this button will sunset that version. This will also sunset any reports completed using that version.

Recommended Version Column Details

• P0 requirement: If a version of the plan is not in the recommended phase, shows the string "None Yet"
• If there is a version currently in the recommended phase, shows:
  • P0 Requirement: Link with text "VERSION_STRING" that targets the single-page view of the plan. Example: V22.12.08
  • P0 Requirement: Show string "Approved DATE" where DATE is the date that the currently recommended version was advanced to the recommended phase.
  • P1 requirement: Show a button that opens the test plan report status dialog. The button label and dialog are described below.

Default Table Sort

• First sort by overall status descending: recommended plans, then candidate plans, then draft plans, then R&D complete plans.
• Then sort by test plan name ascending.
• Then sort by Covered AT ascending

Why?

• In the near term, before filters are implemented, this will put the most important admin work near the top.
• In the long term, This will show all users an alphabetical list of all recommended plans at the top and provide access to the history of those plans.

Links to single-page view of test plan

Note that we will format version identifiers as "Vyy.mm.dd" where yy.mm.dd represent the date of the commit that landed changes to the plan.

This view of the alert plan is provided as an example of a single-page view of a plan. Note that:

1. The link needs to be able to target a specific commit. The version of the plan that is "R&D Complete" will always be at the head of the main branch. However, if there is an earlier version of that plan that is in a later phase, it will be at an earlier point in the revision history.
2. There is a last edited date for each test that is not accurate.
   • It is showing the date and time of the latest commit to main instead of the date and time of revision of that plan.
   • The position of the last edited date implies that we can track when each test was edited, but we can't. We only know the date of the plan revision.
3. Changes to that single-page view should include:
   • Remove last edited from the individual tests.
   • Add an H2 with "Version created on DATE_TIME" under the H1 plan title.
   • Add a paragraph with "Summary of change: COMMIT_TITLE" followed by a div or paragraph containing the commit description.

Confirmation prompts for advancing a version from one phase to next

P0 Requirement: When a button is activated for advancing a test plan version to the next phase, a confirmation prompt is provided. The prompt must include the name of the test plan, its version, the covered AT, and the name of the new phase.

required Reports Definition

To provide a brief, easy to understand reporting status that facilitates the job of shepherding a specific version of a test plan through the working mode process, this design proposal introduces the concept of required reports. A version of a test plan will be required to have reports for specific AT/browser pairs depending on the working mode phase of the test plan version and the set of AT covered by the plan. If a required report has not been completed or is not in progress, the system will show it as "missing".

The draft and recommended phases will each require reports using a specific set of AT/browser pairs. At this time, the pairings for which reporting is required are as follows.

Phase	AT	Browser
Draft	JAWS	Chrome
Draft	NVDA	Chrome
Draft	VoiceOver for macOS	Safari
Recommended	JAWS	Chrome
Recommended	JAWS	Firefox
Recommended	NVDA	Chrome
Recommended	NVDA	Firefox
Recommended	VoiceOver for macOS	Safari
Recommended	VoiceOver for macOS	Chrome

When we have sufficient testing capacity, the next change will be adding recommended phase reporting requirements for Edge on Windows.

For each AT, the system will need to map valid AT/browser pairings as shown in the above table. Thus, it will not expect a plan that includes VoiceOver for iOS to generate a report on VoiceOver for iOS with Safari for macOS. Similarly it would not expect a plan that includes JAWS to generate a report on Jaws with Safari.

A report for a specific AT/browser pairing will be classified as "missing" if it meets any of the folowing conditions.

Condition Description	System Logic
Draft plan report generated with a required AT/browser pair has not been completed and is not in progress	Plan version is in draft phase AND given AT is covered by plan AND given browser is required in draft phase AND report on given AT/browser pair generated using any versions of given AT/browser pair is not complete AND test plan run for given AT/browser pair is not in queue
Recommended plan report generated with latest versions of a required AT/browser pair has not been completed and is not in progress	Plan version is in recommended phase AND given AT is covered by plan AND given browser is required in recommended phase AND report on given AT/browser pair generated using latest versions of given AT/browser pair is not complete AND test plan run for given AT/browser pair is not in queue
Recommended plan report generated with latest versions of an AT/browser pair that is not required but has at least one existing report has not been completed and is not in progress	Plan version is in recommended phase AND given AT is covered by plan AND given browser is not required in recommended phase AND plan has a complete report for any version of given browser AND report on given AT/browser pair generated using latest versions of given AT/browser pair is not complete AND test plan run for given AT/browser pair is not in queue

Note that in draft phase it is up to the admin to decide whether the AT and browser versions are sufficiently recent. This is because it can be difficult to ensure that all people running manyual tests are running the same versions, and the latest versions are not always necessary for candidate review. However, for recommended reports, the objective is for the system to help us keep reports current using latest versions. For reports on recommended plan versions, either the admin will need to seek people who have the necessary versions or the automation system will need to provide the report.

• P0 Requirement: Database includes mapping of valid AT/Browser pairs.
• P0 Requirement: Database maps which browsers have reporting required for draft and recommended phases.
• P0 Requirement: Database tracks latest versions of AT and browsers.
• P2 Requirement: UI enabling admins to modify above mappings.

Test plan report status dialog Description

This dialog shows detailed status of AT/browser reports for a single version of a single test plan.

The label on the button that opens the dialog provides information about the current report status for the test plan. The button has one of the following labels.

Button Label	Condition
Required Reports Not Started	For every AT/browser pair that requires a report, the report is neither complete nor in the test queue.
Some Required Reports Missing	At least one of the AT/browser pairs that requires a report neither has a complete report nor has a run in the test queue. At the same time, at least one of the AT/browser pairs that requires a report either has a complete report or has a run in the test queue.
Required Reports In Progress	At least one AT/browser pair that requires a report does not have a complete report and is in the test queue. All other AT/browser pairs that require a report are either complete or are in the test queue.
Required Reports Complete	All AT/browser pairs that require a report have a complete report.

The dialog title is: "Report Status for TEST_PLAN_NAME Test Plan"

The dialog content includes a note and a table.

• Note content:
  • "Note: This plan is in the PHASE_NAME phase. N AT/browser pairs require reports in this phase."
  • Example: "This plan is in the Draft Review phase. 3 AT/browser pairs require reports in this phase."
• The table is captioned:
  • "Reports for PHASE_NAME TEST_PLAN_NAME Test Plan"
  • Example: "Reports for Draft Alert Test Plan".

The table columns are:

1. required?: Shows "Yes" or "No"
2. AT:
   • If phase is draft, shows AT name.
   • If phase is recommended, Shows AT Name and version.
3. Browser:
   • If phase is draft, shows browser name.
   • If phase is recommended, shows browser name and version.
4. Report Status:
   • If the report is complete, shows a link to the report with text "Report Completed on DATE"
   • If report is missing, shows the string "Missing", and if user is admin, button "Add to Test Queue"
   • If report is not complete and not missing and not in test queue, shows the string "None", and if user is admin, "Add to Test Queue" button
   • If report is in the test queue, shows the string "X% complete by Y testers with Z conflicts"
   • Note: Percent complete = NUMBER_COMPLETED_TESTS_BY_ALL_TESTERS / (NUMBER_ASSIGNED_TESTERS * NUMBER_TESTS_IN_PLAN) * 100

The table is sorted by required Yes, then required No, then AT, then browser.

The rows that appear in the table are determined as follows.

• If the phase is draft, there is one row for each valid AT/browser pair for all AT covered by the plan.
• TODO: If the phase is recommended, the algorithm for determining what rows to put in the table will be described in the next update of this specification.

Filters

Adding a set of quick filter buttons above the table is a P2 requirement. the label on Each filter will show the filter name and number of table rows it will render.

The filter buttons are:

1. All Plans
2. R&D Complete
3. In Draft Review
4. In Candidate Review
5. Recommended Plans

Note: Because each plan may have multiple versions moving through the process, i.e., a version active in more than one phase at the same time, the set of rows shown by each filter is not unique. Thus, if the label on the all plans button were "All Plans 65", the sum of the numbers shown in the labels of the other buttons will likely be greater than 65.

[mcking65]

There is some missing content in the top comment that describes this issue. I am working on it and will update the top comment as I finish each section.

[mcking65]

@jscholes @jugglinmike @ccanash @lolaodelola @isaacdurazo

I've now completed documentation in the top comment. Since Thursday, in addition to completing the missing sections, I have made a number of other changes to either simplify or clarify the requirements.

I realize that this could be a visually wide table with seven columns. I feel pretty strongly that all 7 columns are necessary and that it is best to avoid combining columns. I've found the column combining we have in other places, like candidate page, to be very problematic as a screen reader user. It also gets in the way of sortability.

If table width is a problem, then let's have an ideation session to specifically address the issue. I have a few ideas on how to visually abbreviate, but I'm also concerned about clarity. It might be feasible to combine name and overall status without too severely hurting readability for SR users and preserving the ability to sort by either, but it would add complexity. So, I hope we can make it all fit without doing any of that.

BTW, for the test plan versions, we probably don't need the full version string with timestamp; date is probably sufficient in the context of this table. The downside is that it would create inconsistency with how test plan version is displayed elsewhere, so again, probably best to avoid that truncation if we can.

[mcking65]

@lolaodelola @ccanash @jugglinmike

I am fairly confident that the P0 and P1 requirements in this proposal will cover close to all if not all MVP gaps we will end up identifying in the working mode scenario analysis. I am anxious to complete the analysis and test this hypothesis.

I believe there are some distinct sets of requirements that could be made into issues that could be added to the MVP project and put into the sprint plan.

One approach to grouping requirements into specific deliverables might be to create issues for each of the following:

• Provide overall status in test plan management table
• Provide links to active test plan versions in phase status columns of test plan management table
• Provide advance to next phase buttons in phase columns of test plan management table
• Provide static phase completion strings in phase columns of test plan management table
• Provide covered AT information in test plan management table
• etc

I don't know if the above approach for segmenting work is useful. I suggest it because I think it might be a good way to prioritize work to deliver the most needed functions first. However, there may be sound technical reasons for using a different approach to dividing deliverables across sprints.

[lolaodelola]

Thanks for this @mcking65! cc @howard-e

[lolaodelola]

@mcking65 I've moved the "Test Plan Report Status Page or Dialogue" information to a separate issue (#649) so that we can have a separation of concerns.

[boazsender]

I just had a chance to review this, and I do think the table would be too wide. Since it's not tabular data, maybe we should use a different UI idiom (e.g. a card for each test plan)? What do you think @isaacdurazo?

[mcking65]

@lolaodelola @ccanash @jscholes

I edited the top comment to:

• Filter section: Clarify description of row number badges in filter labels
• All phase status columns: Provided example link to single-page view of plan
• Added a section describing the single-page plan view and changes that are needed to the existing single-page view
• Revised wording of required reports description to remove implication that we need to track operating systems

I will next work on a mock up of the table data.

[mcking65]

@ccanash wrote:

Hi @mcking65, once you have answered those questions we would have a mockup for the table. Since you provided the HTML version of the table I wanted to ask you if it's ok for us to move forward with the implementation or would you like for us to add a description of the visual details that Isaac has designed for your approval and share the mockups here on Github?

Both:

• Move forward with implementation
• Share mockups and descriptions as soon as available
  I assume that even if we need minor tweaks to design that won't have much if any negative impact on on already completed implementation work. That is asuming that the mockups and descriptions are nearly complete.

[isaacdurazo]

Visual Mockup

Here are the visual details for each column from the mockups and styling proposed for the Status Summary Table on the Data Management page.

Test Plan Column

Test plan names are in bold

Covered AT column

All strings are in plain text

Overall Status column

• All status strings "PHASE Since DATE" have been visually split into two styles, meaning one of these two will need to be in its own HTML container, such as a <span> so we style them differently. "PHASE" is inside a visual container with rounded corners and each phase uses a different background color. "Since DATE" is in small plain text and appears underneath "PHASE". The colors used for each phase are:
  • Draft: Gray
  • R&D: Blue
  • Candidate: Orange
  • Recommended: Purple
• The String “+N New Versions in Progress" is in smaller text and has its “+N” wrapped in a gray container similar to the status label, but looks more like a notification than an actual label. The rest of the string is in plain text. This string is all in one line.

R&D Version, Draft Review, Candidate Review, and Recommended Versions columns

• String “N/A” is in small gray text and it’s centered in the cell.

• “Advance to next phase” button uses the style we have for secondary priority buttons, which is black text on a white background and a gray outline.

• Link with the text "VERSION_STRING" is in blue and has a green checkmark icon on its left. These “VERSION_STRING” texts have a light gray background to set them apart from the rest in the cell but are not as prominent as the “Advance to next phase” buttons.

• String “Review of VERSION_STRING completed DATE” is now “VERSION_STRING Review Completed DATE”. “VERSION_STRING” uses a similar style to the Link with the text "VERSION_STRING”, but in this one, the checkmark is gray instead of green and the text is plain black since it’s not a link.

• String “N Open Issues” is in small plain text and has a yellow circle on the left as an icon to represent the concept of “issues”

• Strings “Target N Days Away” and “Target N Days Past” are in small plain text.

• Test plan report statuses

  • Required Reports Not Started: small plain text with a gray circle on the left as an icon.
  • Some Required Reports Missing: small plain text with a red circle on the left as an icon.
  • Required reports in Progress: small plain text with a blue circle on the left as an icon.
  • Required Reports Complete: small plain text with a green circle on the left as an icon.

[isaacdurazo]

This is an updated mockup with P1 features for the Status Summary Table on the Data Management page. Here is this summary of updates for each column:

1. Test Plan - The test plan name is now a link
2. Covered AT - Covered categories such as "3 Desktop Screen Readers" are now displayed as buttons
3. Draft Review - Report status text such as "Required Reports Complete" is now displayed as a button
4. Candidate Review - Timeline string is a button
5. Recommended Version - Report status text such as "Required Reports Complete" is now displayed as a button

[isaacdurazo]

Here is a mockup that includes filtering and sorting

Text-based Mockup

Filter

This mockup includes a Filter button at the top right of the Test Plans Status Summary table, in the same row we have the “Test Plans Status Summary” heading. Clicking this button will display a dropdown menu with all the filtering options. This menu uses checkmarks for selecting filters, which means multiple filters can be selected at once, except for the “All Plans” filter.

When the filter is active but closed, the filter label will display a number with the number of filters selected.

Sorting

There are three sortable columns: Test Plan, Covered AT, and Overall Status.

Sortable columns will display a pointing down arrow icon on hover. If the column is in its default order, when clicked, it will display a dropdown menu with two sorting options: Ascending and Descending. If one of the sort options is already selected, the items in the drop-down menu are: Clear Sort and Reverse Sort Order

When a column has been sorted by any of the two options, the corresponding icon will be persistent next to the column heading. Clicking this icon is another way of reversing the sort other in addition to doing it from the dropdown menu

Visual Mockup

[mcking65]

@isaacdurazo wrote:

This mockup includes a Filter button at the top right of the Test Plans Status Summary table, in the same row we have the “Test Plans Status Summary” heading. Clicking this button will display a dropdown menu with all the filtering options. This menu uses checkmarks for selecting filters, which means multiple filters can be selected at once, except for the “All Plans” filter.

I have to apologize because One thing I neglected to include in the original proposal is that I saw the filters as serving two separate and distinct but equally important functions.

Certainly one use case is filtering. But another that is just as important is to have a persistent at--a-glance view of our most important top-line numbers related to test plan development and management. That is, to summarize the state of the project with the number of patterns with plans, how many of them have an active R&D version, how many have a version in draft review, how many have a version in candidate review, and how many have a recommended version.

Providing that top-line summary of the status of ARIA-AT is the reason I suggested a design with separate filter buttons for each phase. I should have said so explicitly. Putting those numbers in a popup menu does not make them visible enough.

I don't know that the use case of choosing multiple filters within this set is particularly important. However, it could still be done with a multi-button approach. In that case, the buttons are independent toggles, except for the All button. If they work that way, then it would also be necessary to add a separate message that says how many rows are displayed.

Sortable columns will display a pointing down arrow icon on hover.

Please provide a persistent icon that indicates sortability. We run into keyboard and cognitive accessibility problems if it is necessary to hover to know that a column is sortable.

If you have a persistent icon on all sortable columns, you may be able to avoid the need for a popup menu. If a column is not sorted, you could make the first click do ascending and the second click do descending. Since this is a small table, I am assuming that latency of sort is low, so the downside of performing an unnecessary sort if the user wants descending would not be an issue. Users would end up performing the same number of clicks, and we could avoid the complexity of the menu, which would be very desirable for screen reader users.

[isaacdurazo]

Thanks for clarifying @mcking65 Here is an updated mockup and its description.

Filter

I'm incorporating a multi-button approach. Between the "Test Plans Status Summary" heading and the table summary, there's a row that starts with the word "Filter" followed by a button for each of the phases. Each one of them includes the number of rows within parenthesis. I think we can start by having the ability to only choose one a at a time since we are not sure about a use case for selecting multiple ones.

Sorting

All three sortable columns have now a persistent icon. This icon has three states: inactive, active ascending and active descending. By default, the Overall Status column will have the ascending state (icon) active, which uses the order you suggested. R&D at the top first and Recommended at the bottom end.

This new approach removes the option to "remove the sorting", so there will a column that's sorted in some way. Please let me know what you think and if this updates reflect your vision.

[ccanash]

Designs have been approved. Closing