Scoring criteria for Interop 2023 Accessibility Testing Investigation

[zcorpan]

We should decide how to measure success for this investigation area.

Scoring criteria:

• Consensus on testing approach and implementation plan (e.g. extend testdriver.js for role/label) (20%)
  • Testing plan and file locations for Interop 2023 Accessibility Investigation #1
  • Accessibility Testing Approach #2
• Computed role and label are testable in wpt infrastructure (20%)
  • Update test driver to use computedrole/computedlabel from webdriver #5
• WebDriver computed role and computed label tests run (not necessarily pass) in
  • Chrome (10%)
  • WebKit (10%)
  • Gecko (10%)
    • Gecko bug 1585622 (webdriver computedlabel/computedrole updates)
    • Gecko bug 1832148 (marionette updates)
• Priority 1 tests in Scoring criteria for Interop 2023 Accessibility Testing Investigation #3 (comment) are written and landed (25%)
  • P1 Role Tests (computed roles based on ARIA and HTML-AAM rules)
    • Role Rules from the ARIA Spec (Fallback Role, Host Language Conflict, Invalid and Abstract Roles, etc.) #8
    • Need role synonym tests for image/img and none/presentation #10
    • fallback roles tests #11
    • abstract role tests #12
    • HTML-AAM computedrole tests #13
    • grid role tests #14
    • listbox roles tests #15
    • menu roles tests #16
    • tab roles tests #17
    • tree roles tests #18
    • invalid roles tests  #19
    • contextual role invalidations tests #20
    • modify generic role tests for HTML-AAM to allow "generic", "none" (Chromium), or "" (WebKit) according to related ARIA spec change #48
  • P1 Label Tests (ARIA AccName "Name Computation" algorithm)
    • AccName tests v1 #27
    • Add basic accname tests for shadow DOM wpt#36541
    • AccName: tests/todos in accname/name/comp_embedded_control #31
    • AccName: tests/todos in accname/name/comp_hidden_not_referenced #32
    • AccName: tests/todos in accname/name/comp_host_language_label #33
    • AccName: tests/todos in accname/name/comp_label #34
    • AccName: tests/todos in accname/name/comp_labelledby #35
    • AccName: tests/todos in accname/name/comp_name_from_content #36
    • AccName: tests/todos in accname/name/comp_text_node #37
    • AccName: tests/todos in accname/name/comp_tooltip #38
• Priority 2 tests in Scoring criteria for Interop 2023 Accessibility Testing Investigation #3 (comment) are written and landed; browser-internal computedrole only not platform-specific role mappings (5%)
  • graphics-aria role mappings needed as precursor to svg-aam role mappings #9
  • Write tests for SVG-AAM role mapping #69
  • Write tests for DPUB-AAM role mapping #71

[cookiecrook]

Another potential to consider is… Can we agree on a list of other inspectable properties that would be beneficial to test in future Accessibility tests?

There is likely some low-hanging fruit that would be non-controversial I think, such as the "checked" state of an Accessibility representation, which would return true for either <input type="checkbox" checked> or <div role="checkbox" aria-checked="true"> There are probably dozens of other inspectable properties in this category.

There are some others that may get us further toward a long-term accessibility tree synchronicity (like getAccessibilityParent and getAccessibilityChildren) because they would make the tree differences more apparent.

A few examples where I expect the DOM-based vs Render-based trees to be different are:

• interactive but mock object children like scrollbars in views with overflow:scroll;
• generated content like ::before and ::after psuedos, including implicit bullets on lists
• possibly positioning containers or ancestry on absolutely positioned views, such as table headers on a "scrolling" table widget.
• dozens more

Although implementation differences there should not count against an implementation, having getAccessibilityParent and getAccessibilityChildren would allow us to catalog the differences and prioritize any potential alignment in future investigations.

[zcorpan]

Those things seem good to make progress on. It seems like "AOM Phase 4" but in webdriver.

How far along for parent/children and states+properties can we expect to get this year, to make full score achievable? Or should we treat it as something that we can work on in parallel from intrinsic motivation without impacting the score?

[cookiecrook]

IMO the latter.

[cookiecrook]

Consensus on Priority from the Kickoff #4

Priority 1 (Scored 25%)

• AccName Label Tests
• Role Rules from the ARIA Spec (Fallback Role, Host Language Conflict, Invalid and Abstract Roles, etc.)
• HTML-AAM Role Computation (Browser Internal Role, not Platform-Specific Mappings)

Priority 2 (Scored 5%)

• SVG-AAM Role Computation (Browser Internal) and resolve precursor issue for Graphics-ARIA
• Potentially a portion of DPUB-AAM Role Computation (Browser Internals only) [see update below]

Investigations (Unscored, 0%)

• What other WebDriver accessibility properties are needed? (checked, valuetext, etc.)
• Computed Accessibility Parent/Children (precursor to internal tree dump?)
• Tree-Dump Explorations (Internal Accessibility Tree and/or Platform Accessibility Tree)
• Core-AAM (Platform-Specific Role Mappings)…
• Remainder of DPUB-AAM Role Computation (Platform-Specific Role Mappings) [see update below]

Please edit directly or comment if I missed something.

[zcorpan]

Thanks, I've updated OP to reflect that scoring.

[zcorpan]

As @gsnedders pointed out in the meeting, there's no need for a wpt RFC to add standard webdriver APIs to testdriver.

And web-platform-tests/wpt#38758 is now merged. 🥳

So maybe we can remove the RFC bits from the scoring criteria, and replace with "testdriver.js implements get_computed_role and get_computed_label (40%)"

[cookiecrook]

DPUB-AAM Role Computation (Browser Internal)

Now that I think about this some more, the original release of DPUB-AAM could have been tested with computedrole (these were just synonyms of ARIA Core roles) but the current version has many platform-specific variants. Some or all of the DPUB-AAM testing may need to be in the Investigations section alongside Core-AAM. I updated above to reflect that change.

[cookiecrook]

@zcorpan wrote:

So maybe we can remove the RFC bits from the scoring criteria, and replace with "testdriver.js implements get_computed_role and get_computed_label (40%)"

Makes sense to me...

RFC is written with implementation plan (e.g. extend testdriver.js for role/label) (10%)

I suppose we could count #1 as that, though less formal and predated the Kickoff.

RFC is accepted (10%)

#2

RFC is implemented (20%)

#5

In any case.... 40%. 🎉

[zcorpan]

I've replaced the RFC bullet points with:

• Consensus on testing approach and implementation plan (e.g. extend testdriver.js for role/label) (20%)
• Computed role and label are testable in wpt infrastructure (20%)

[zcorpan]

Bugs for the next milestone:

• https://bugs.chromium.org/p/chromium/issues/detail?id=1422189
• https://bugs.webkit.org/show_bug.cgi?id=253500
• https://bugzilla.mozilla.org/show_bug.cgi?id=1585622

[cookiecrook]

I agree we should diagnose those failures, but I don’t think they block any work. Here are the new tests that were merged yesterday.

https://wpt.fyi/results/wai-aria/role/basic.html?label=master&label=experimental&aligned&view=subtest
https://wpt.fyi/results/accname/basic.html?label=master&label=experimental&aligned&view=subtest

[zcorpan]

I think it would be more convincing that we have a solid foundation if the get.py tests also pass. But maybe fixing those failures could be considered for next year's focus area, if we're able to write new tests anyway.

I can change the scoring criteria to link to the basic.html tests instead.

[gsnedders]

I think it would be more convincing that we have a solid foundation if the get.py tests also pass. But maybe fixing those failures could be considered for next year's focus area, if we're able to write new tests anyway.

As I said on https://bugs.webkit.org/show_bug.cgi?id=253500, I don't think all the error handling tests are of equal priority for enabling accessibility testing (and test_no_browsing_context and test_no_such_element tests fail all over the place for both Chrome and Safari). I think we only need to care about the test_get_computed_label and test_computed_roles tests themselves.

[cookiecrook]

What's the process for indicating on https://wpt.fyi/interop-2023 that the investigation is already at 60%?

[zcorpan]

I think a PR in https://github.com/web-platform-tests/wpt.fyi

Example: web-platform-tests/wpt.fyi@6c32b12

[zcorpan]

I also see interop-data_v2.json, I'm not sure why there are two files. cc @foolip @DanielRyanSmith

[DanielRyanSmith]

I also see interop-data_v2.json, I'm not sure why there are two files.

This was a quick fix on the day of the Interop 2023 dashboard release. There was a GCP caching issue related to this file being stored in the static folder that caused some users to see a broken page on the dashboard. The file was copied with a new name to avoid serving a cached version. As of today, the file with the _v2.json suffix is the one being referenced in production, and this file needs to be moved out of the directory it lives in to avoid possible caching problems, which is what is captured in the linked issue.

Edit: I added a PR to remove the old interop-data.json in order to avoid confusion.

[zcorpan]

What's the process for indicating on https://wpt.fyi/interop-2023 that the investigation is already at 60%?

web-platform-tests/wpt.fyi#3207

[cookiecrook]

Gecko bug 1585622 was resolved today. 🎉

[zcorpan]

Yay! It will be part of geckodriver 0.33.0 (bug 1814050).

[zcorpan]

geckodriver 0.33.0 has been released, but some more glue is required for this to work on wpt.fyi: https://bugzilla.mozilla.org/show_bug.cgi?id=1827934

[cookiecrook]

.25*(11/23) + .05*(1/3) = 0.1362318841

So 70% + 13.6% = ~84%

[cookiecrook]

If we want to add the currently-unscored-but-high-percentage-of-passing-results to the dashboard, that link is live now using the new accessibility label.

• 87%: WebKit and Chrome.
• 84%: Gecko.
• 78%–86%: Edge (ambiguous, b/c it skips the crashtests, so is running 427 rather than the full 466)

[cookiecrook]

FYI @benbeaudry 👆

[cookiecrook]

Update. There are a more than 466 total tests... Some are being masked from the results due to:

• Directory labels are ignored when a file-specific label is applied wpt-metadata#4665

[cookiecrook]

.25*(11/23) + .05*(1/3) = 0.1362318841
So 70% + 13.6% = ~84%

The following three are approved and ready to merge, and would each bring the total up 1 percentage point once merged. 85% for 1, 86% for 2 more, 87% when all 3 are merged.

• menu roles tests #16
• tab roles tests #17
• tree roles tests #18

[cookiecrook]

@KyleJu is steadily knocking out the search bugs on wpt.fyi, so I think we have our first accurate count of the accessibility tests in WPT, currently at 586! Most of them newly created this year!

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility

[cookiecrook]

Related TPAC breakout time scheduled:
https://www.w3.org/events/meetings/7ea3022f-4424-45da-ba98-2953baf7d997/

[zcorpan]

Submitted focus area proposal for interop 2024: web-platform-tests/interop#526

Feel free to comment there with additional info. Thanks!

[cookiecrook]

89% as of today thanks to @adampage for a merged PR and @jnurthen for a review!

Several more reviews outstanding if others want to help us hit 90%

[cookiecrook]

Over 90% now. A few more rolling in this week. Will update the dashboard score once those land.

[cookiecrook]

All roles tests complete. Waiting on 2.5 PRs to merge for AccName, which will put us at 96.667%

[cookiecrook]

All P1 tests complete, so we're at .95 + .05*⅓ ≅ 97%

The DPUB PR is complete but not yet landed, so with that caveat, we're at .95 + .05*⅔ ≅ 98% (score update PR)

Final 2% in the SVG tasks on me. (WPT PR #43196)

[cookiecrook]

💯