Trend reports design proposal

[boazsender]

This issue describes a design for trend reports to meet the following requirement from the project readme:

Access to reports on specific AT/browser versions: Minimal UI changes that enable publication of reports for a specific AT/browser version and visibility of reports by AT and browser version.

This design proposal consists of two tables outlined below.

1) Trend reports table

The trend reports will consist of a summary table of all historical runs, where each row links to a detail table. This summary table will take the place of the existing table at https://aria-at.w3.org/reports which currently displays the lates trun.

Summary table heading (column names):

• Test plan version (git sha of test suite)
• One column for every AT/Browser combo
• Date of run

Summary table rows

• One row for each run
• A check mark in the cell for each AT/Browser combo if the combo was run with a link to that run

2) Detail table

The detail table will be the same as the current table that exists at https://aria-at.w3.org/reports, but reproduced for every historical run and moved to a url dedicated to the git sha of the given run (e.g. https://aria-at.w3.org/reports/runs/$SHA)

[mcking65]

I don't understand the base concept underpinning this proposal. Perhaps I have a fundamentally different idea of "trend" in this context. What I have meant by "trend" is how the interop changes from version X of an AT to version x+1, x+2, ... of that AT for a given AT/Browser/test plan. Version x is the first version of an AT for which we have a report generated from the recommended version of a test plan.

With this "concept of trend", the test plan version and run date are not relevant dimensions of the trend data. The only relevant dimension for any given AT/Browser combination is the version of the AT. In the future, we might also show a trend by browser version. It remains to be determined whether or not browser version trends have any value.

For now, I think we should keep the top-level page as is so that it always shows data for the latest versions of an AT that has been tested. That is still the most important data to surface.

One light-weight approach to providing access to data about all versions of an AT that have been tested for a given test plan might be to provide a AT versions summary table at the top of each detailed report page. So, users would continue to see the latest data on the top-level page and on the second-level page. But when they get to the detailed report on a test plan for a given AT/browser combination, they would see a summary of all versions of that AT tested for that plan.

For example, if you are on the reports page and follow the link for the Alert Example, you get to a page showing a summary of the latest data for each AT/browser combination that was tested. Then, if you activate the "View Complete Results" button for VoiceOver and Safari, you get to the detailed report page for that combination. Currently, that page title does not include a VoiceOver version. In the future, if the test plan is in the recommended phase, the page title would include the version of VoiceOver. At the top of that page, we could have a VoiceOver versions summary table showing a summary of data for all versions of VoiceOver that have been tested. It could have columns for VoiceOver version, MUST HAVE Behaviors, SHOULD HAVE Behaviors, and MAY HAVE Behaviors. Users could see how VoiceOver support for that test plan has changed from version to version by reading down the columns. Each version could be a link to the detailed report for that version.

Side note: it would be helpful if each of the detailed report pages included a copy of the type of summary table that is shown at the second level in the IA. For example, it would be helpful if the VoiceOver for macOS and Safari for Alert Example report page included a copy of the VoiceOver and Safari table from the Alert Example report page.

[boazsender]

Sounds good matt. Lets take that approach.

[isaacdurazo]

Here's a mockup that matches your proposal including the side note about a copy of the type of summary table that is shown at the second level in the IA. Cc @ccanash

[ccanash]

Closing as completed, we will start implementation in #1083