Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Accessibility (computed role + computed label) #526

Closed
3 tasks done
zcorpan opened this issue Oct 4, 2023 · 37 comments
Closed
3 tasks done

Accessibility (computed role + computed label) #526

zcorpan opened this issue Oct 4, 2023 · 37 comments
Labels
focus-area-proposal Focus Area Proposal

Comments

@zcorpan
Copy link
Member

zcorpan commented Oct 4, 2023

Description

Fix interop bugs in the browser’s accessibility tree for the parts that are now testable in wpt as a result of the interop-2023 accessibility testing investigation area, for the benefit of users who need assistive technology and for web developers wishing to design accessible user interfaces on the web.

Specification

https://w3c.github.io/aria/ https://w3c.github.io/accname/ https://w3c.github.io/html-aam/

Open Issues

No response

Tests

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility

More specifically, the “priority 1” tests in web-platform-tests/interop-accessibility#3 (some tests are open PRs, some are not yet created but should be soon). Exclude “priority 2” (graphics role, SVG, DPUB).

Current Implementations

  • Blink
  • Gecko
  • WebKit

Standards Positions

No response

Browser bug reports

No response

Developer discussions

No response

Polls & Surveys

in preliminary results from State of HTML 2023, ARIA was a common response to the question "Which existing HTML features or browser APIs are you unable to use because of browser differences or lack of support?"

Also see accessibility_pain_points responses in the same survey: https://coda.io/d/State-of-HTML-Preliminary-Results_d7UVQ_fZW0V/Other-Core-Pain-Points_suDnF#Other-Core-Pain-Points_tuN-h/r4723

Inconsistent handling of elements/components between screen readers & browsers

As a FOSS developer, I can only test for browser/OS/AT combinations I personally own and that doesn't give me enough confidence

Most of the problems are UA problems - the UAs need to be quicker to fix bugs, so that developers who implement the specs can be confident they aren't doing the wrong thing

Browser support

Segmented support between browsers and OSes, have hit many cases I can support in 2 browsers but not a third. Safari and Firefox are the usual culprit.

Inconsistency between AT and browsers on how information is exposed.

Differences between browsers and tools (e.g. NVDA vs VoiceOver)

Difference between browsers, OS and screen readers

browser/AT interop

Different browsers expose different information to different assistive technology tools in different ways. Testing accessibility is a huge burden due to the discrepancies between the AT

Existing Usage

No response

Workarounds

No response

Accessibility Impact

Positive; this is explicitly designed to improve interop of some of the most important browser accessibility features.

Privacy Impact

No response

Other

No response

@foolip
Copy link
Member

foolip commented Oct 5, 2023

For survey data and web developer demand, in preliminary results from State of HTML 2023, ARIA was a common response to the freeform question "Which existing HTML features or browser APIs are you unable to use because of browser differences or lack of support?"

@dtsengchromium
Copy link

I think there's issues with the tests themselves and the way they fetch data from Chrome. One issue I've written up follows.

WPT calls into webdriverio
https://github.com/webdriverio/webdriverio
in
webdriverio/packages/devtools/src/commands/getElementComputedRole.ts
accessibility.snapshot

which is Puppeteer
https://github.com/puppeteer/puppeteer
puppeteer/packages/puppeteer-core/src/cdp/Accessibility.ts
which sends the CDP command
Accessibility.getFullAXTree
with the interesting option defaulted to true.

In Chrome,
getFullAXTree
InspectorAccessibilityAgent::getFullAXTree
gets called and returns the tree back over CDP.

Back in Puppeteer,
puppeteer/packages/puppeteer-core/src/cdp/Accessibility.ts
Accessibility.snapshot

the tree is filtered for "interesting nodes". This amounts to calling into its own wrapper AXNode ts object isInteresting method.

That method has some definition of interesting which includes only some elements e.g. controls, leaves (with text), etc. That explains a lot of failures within the dashboard pertaining to roles.

I'm therefore not sure we can use the test results here until the underlying tooling is reliable/corrected. Specifically here, the simplest thing is for WPT to pass interesting to false
https://pptr.dev/api/puppeteer.accessibility.snapshot

@cookiecrook
Copy link

cookiecrook commented Oct 10, 2023

@dtsengchromium The full AX tree dump investigation is not part of this proposal, so unless I'm misunderstanding, your comments would NOT block this Interop 2024 Focus Area proposal… Is that correct? cc @aleventhal

@cookiecrook
Copy link

Though I could see ARIA Reflection (Gecko bug 1858211) being a problem for Gecko if there are objections to the test expectations or spec. If so, we could exclude that test from the Focus Area too.

@jcsteh
Copy link

jcsteh commented Oct 11, 2023

[Edited by James: Oops, I accidentally corrected this one when trying to correct my own above. Sorry, this quotation is back to the original now. - jc]

@jcsteh The full AX tree dump investigation is not part of this proposal, so unless I'm misunderstanding, your comments would NOT block this Interop 2024 Focus Area proposal… Is that correct?

I don't see any comments from me on this particular issue, so I assume you're referring to web-platform-tests/interop-accessibility#51 (comment)? So no, those comments don't block this because it's out of scope.

You're correct that ARIA element reflection is a problem for us with regard to this focus area. This should be discussed somewhere else, but as I understand it, one of the primary use cases of that entire spec is to support references across shadow DOM, yet it doesn't actually achieve that goal because of the encapsulation problem.

There are a few other tests I think we should consider excluding from this as well for various reasons, but I'll provide more details on that once I've finished my triage.

@jcsteh
Copy link

jcsteh commented Oct 11, 2023

There seem to be some tests where we don't have a clear path forward and no engine is passing. For example, html-aam/fragile/area-role.html and el-thead/el-tfoot/el-tbody in html-aam/table-roles.html. We'll either need to exclude these or collectively determine a path forward in order to get them passing.

@dtsengchromium
Copy link

@cookiecrook

My comment above was referring to the way in which trees were fetched for the tests and went into detail on the manner WPT does it. Happy to have that corrected if I missed something. The takeaway is that many of the results don't mean anything at the moment.

Separate questions are whether webdriver is the right component to base these tests off of and if so, whether its worth making contributions there. For Chrome, webdriver leverages puppeteer. Who wants to make contributions to those projects to get things working? IIRC, those projects primarily concern themselves with test automation.

Also, note that puppeteer is fetching Chrome's internal Blink side a11y tree. This is probably not the right tree to test against. For example, the role literals are internal programmatic identifiers and not stable. The name computation is incomplete since we do different things for each platform a11y api (grant it, in the minority of cases).

The platform trees are the only stable representation which we really service (e.g. NSAccessibility, IAccessible2, etc). These questions and others worth a discussion at the next meeting for a11y interop.

@boazsender
Copy link
Contributor

boazsender commented Oct 12, 2023

I agree with @cookiecrook that web-platform-tests/interop-accessibility#51 is not part of this proposal as it's written. This proposal is about closing interop gaps from WPTs that don't use the axe tree.

@dtsengchromium what you're saying makes sense, we reached the same conclusion at Bocoup when we tried to use the available accessibility tree dump through cdp with the intention of testing core-aam and realized we couldn't for the reason you've mentioned here. I think we should continue this conversation on web-platform-tests/interop-accessibility#51 and discuss the best path forward for exposing the platform trees for testing aam.

I also started an issue here about continuing the 2023 accessibility investigation area into 2024, which is a place where I think we could continue this work if we don't advance it further this year: #459

So the tree dumping conversation would be part of the 2024 investigation area carry over, whereas this issue is a new focus area for closing interop bugs from tests based on things like getcomputedrole. I know it's a lot of interop jargon, so apologies for the category nitpicking. Where these things fall ends up being important for the wpt interop prioritization process. That's the only reason I'm pushing on this detail.

@zcorpan
Copy link
Member Author

zcorpan commented Oct 12, 2023

I think there's some misunderstanding here; as noted in web-platform-tests/interop-accessibility#65 (comment) , @dtsengchromium did not comment about tests using tree dump (which don't yet exist?) but about tests using get computed role (which is implemented using getFullAXTree for Chrome).

@boazsender
Copy link
Contributor

Ah I understand now, thanks for clarifying @zcorpan.

@dtsengchromium what do you think would the best path forward for changing the underlying tooling?

@cookiecrook
Copy link

Yes, thanks for clarifying… FWIW, Chromium is already passing ~92% of the tests proposed for next year, but I can see the HTML-AAM: el-code test is failing as @dtsengchromium describes.

@cookiecrook
Copy link

@jcsteh wrote:

I don't see any comments from me on this particular issue,

Yes, sorry. That was a GitHub autocomplete that I mistakenly confirmed. Apologies as I meant to reply to @dtsengchromium

@dtsengchromium
Copy link

Fixing the tooling would involve upstream working in possibly each of WebDriver, Puppeteer. I don't know if this work could be rolled up into any other efforts/FAs?

@jensimmons
Copy link
Contributor

We should figure out how to define the scope of the tests chosen for Interop 2024, and then set the scope. Not keep letting it grow over the course of the year.

@jugglinmike
Copy link

@dtsengchromium Thanks for that analysis! I wanted to understand the extent of the dependency on webdriverio, and I figured that I'd be able to find it in WPT since most of the "priority 1" tests which @zcorpan references have been completed. I searched WPT for references to webdriverio, but nothing came up:

$ git log -n1 --format='%h %ci %s'
b2528beea2 2023-11-01 11:05:04 -0700 [css-tables] Make `col-change-span-bg-invalidation*.html` tests fuzzy (#42897)
$ grep -ir webdriverio . | wc -l
0
$ find . -iname '*webdriverio*' | wc -l
0

I thought maybe we're talking about the Chromium project's integration with WPT, but I couldn't find any references in Chromium's source tree, either:

$ git log -n1 --format='%h %ci %s'
a87e318028 2023-11-02 00:55:19 +0000 Cleanup "inverse-fieldtrial" trybots.
$ grep -ir webdriverio . | wc -l
0
$ find . -iname '*webdriverio*' | wc -l
0

Would you mind pointing me in the right direction?

@dtsengchromium
Copy link

dtsengchromium commented Nov 2, 2023

@jugglinmike I'm not too familiar with the WPT repo and how it's set up and it's been a couple of weeks. What you're looking for in WPT is the point at which it calls get computed role (I don't recall the exact method name). I traced down a couple of layers in WPT from the test html file to a common test ts file which appeared to lead to web driver.

It does also appear like maybe WPT has support for multiple test harnesses (e.g. servo, selenium, marionette).

@jugglinmike
Copy link

After tracing the flow of control, I've found a couple aspects that differ from what @dtsengchromium reported:

  1. The tests appear to be instructing WPT's test runner to send WebDriver commands to Chromedriver (that is, neither webdriverio nor Puppeteer appear to be involved)
  2. It seems as though the CDP command named Accessibility.getPartialAXTree is in use (not Accessibility.getFullAXTree)

These discrepencies are important because they impact how we'll fix the problem (and plan for the fix). I'll share the complete trace in the hopes that someone can verify my research or more precisely dispute it. (It's even more likely than normal that I've made a mistake because this infrastructure has been built with extension in mind.)

Footnotes

  1. The Chromium project inserts a custom definition of test_driver_internal, but I haven't analyzed that path because we're considering the test results published on wpt.fyi.

@cookiecrook
Copy link

cookiecrook commented Nov 7, 2023

@cookiecrook
Copy link

cookiecrook commented Nov 7, 2023

The fragile tests should be excepted from the Interop Focus Area proposal if web-platform-tests/wpt#39617 is not resolved.

@gsnedders
Copy link
Member

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility

More specifically, the “priority 1” tests in web-platform-tests/interop-accessibility#3 (some tests are open PRs, some are not yet created but should be soon). Exclude “priority 2” (graphics role, SVG, DPUB).

This leaves, what? https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility%20and%20not%20path%3A%2Fsvg-aam%20and%20not%20path%3A%2Fdpub-aam%20%20and%20not%20path%3A%2Fgraphics-aria%20and%20not%20is%3Atentative maybe?

@cookiecrook
Copy link

cookiecrook commented Nov 29, 2023

Perhaps also not path:/html-aam/fragile (I can rename that dir tentative if that's better)... I think it's unclear what the role should return for never-rendered elements like image map > area and the blocking issue is actually on the WPT test harness, not on an implementation.

https://wpt.fyi/results/?label=master&label=experimental&aligned&q=label%3Aaccessibility%20and%20not%20path%3A%2Fsvg-aam%20and%20not%20path%3A%2Fdpub-aam%20%20and%20not%20path%3A%2Fgraphics-aria%20and%20not%20is%3Atentative%20and%20not%20path%3A%2Fhtml-aam%2Ffragile%20

@foolip
Copy link
Member

foolip commented Dec 1, 2023

I've taken a look at the test failures in Chrome to understand #526 (comment) and #526 (comment).

The 28 failing tests are:

/accessibility/crashtests/svg-mouse-listener.html
/accname/name/comp_embedded_control.html
/accname/name/comp_hidden_not_referenced.html
/accname/name/comp_label.html
/accname/name/comp_name_from_content.html
/accname/name/comp_text_node.html
/accname/name/comp_tooltip.html
/html-aam/fragile/optgroup-role.html
/html-aam/roles-contextual.html
/html-aam/roles-dynamic-switch.tentative.window.html
/html-aam/roles-generic.html
/html-aam/roles.html
/html-aam/roles.tentative.html
/html-aam/table-roles.html
/html/dom/aria-element-reflection.html
/inert/inert-inlines-around-selection-range-in-contenteditable.html
/svg-aam/role/roles-generic.html
/wai-aria/idlharness.window.html
/wai-aria/role/contextual-roles.html
/wai-aria/role/fallback-roles.html
/wai-aria/role/grid-roles.html
/wai-aria/role/list-roles.html
/wai-aria/role/listbox-roles.html
/wai-aria/role/menu-roles.html
/wai-aria/role/synonym-roles.html
/wai-aria/role/tab-roles.html
/wai-aria/role/table-roles.html
/wai-aria/role/tree-roles.html

Virtually all of these use aria-utils.js, which internally depends on test_driver.get_computed_role() and test_driver.get_computed_label(). The wpt.fyi Chrome results should end up using the implementation of this in testdriver-extra.js and executorwebdriver.py, which bottoms out in calling https://w3c.github.io/webdriver/#get-computed-role or https://w3c.github.io/webdriver/#get-computed-label.

@dtsengchromium, webdriverio or puppeteer are not involved here, and it's also not related to the work to dump the full accessibility tree. I wonder if there's been a crossing of the wires between this proposal and the investigation effort? I'm not very in touch with the details, but I hope I got the above right. It would be very useful to analyze the Chrome failures and confirm if they're real implementation bugs, or if there are indeed infrastructure issues that need resolving.

@cookiecrook
Copy link

@foolip @dtsengchromium

there is another issue tracking the Chrome-specific stuff here: web-platform-tests/interop-accessibility#65

@cookiecrook
Copy link

cookiecrook commented Dec 5, 2023

Accessibility WPT meeting: Group opinion is that the Chromium implementation issue is not a blocker for this Focus Area. The bug in ChromeDriver (presuming that's where the bug is) can be fixed as part of the effort to improve the Focus Area score.

@cookiecrook
Copy link

cookiecrook commented Dec 6, 2023

The SVG-specific tests were Priority 2 for the investigation and therefore excluded from this Focus Area tests, but of the 26 SVG tests finished for the investigation…

  • WebKit passes 25 of 26.
  • Chromium passes 25 of 26.
  • Gecko passes 17 of 26 (failures mainly in the host lang label tests).

So we could leave it up to Mozilla to determine whether to include those 26.

@jcsteh
Copy link

jcsteh commented Dec 11, 2023

I think it's reasonable to include these. I do have an open question about one of them (also failed by Chromium) in w3c/svg-aam#32 (comment).

@zcorpan
Copy link
Member Author

zcorpan commented Dec 11, 2023

@foolip is including the SVG tests ok for Chromium? (see above)

@jensimmons
Copy link
Contributor

Tests of display:contents for consideration to include: web-platform-tests/wpt#43740

@dtsengchromium
Copy link

dtsengchromium commented Jan 6, 2024

@foolip

Thanks for the analysis! My bad on the web driver, puppeteer paths; wasn't clear where things went in WPT.

It looks like the pruning may occur in several places. Will update here with more next week.

@cookiecrook
Copy link

@dtsengchromium No need to post that update here because it's not relevant to the Accessibility Interop Focus Area this issue is tracking.

There's another issue tracking the ChromeDriver bug in ExecuteGetComputedLabel. Please update that one:

@cookiecrook cookiecrook changed the title Accessibility (computed role + accname) Accessibility (computed role + computed label) Jan 12, 2024
@cookiecrook
Copy link

Minor adjustment to the issue title because not all the label stuff (img alt or html:label for example) that we are testing is defined in the AccName spec.

@cookiecrook
Copy link

cookiecrook commented Jan 12, 2024

As an aside, all the display: contents; accessibility tests from #568 are also only able to confirm testing via computedrole and computedlabel, so those tests could be rolled under this one if desired, as opposed to a separate Focus Area, but it seems fine either way.

@dandclark
Copy link
Contributor

This proposal has been accepted as part of Interop 2024! On behalf of the entire Interop team, thank you for proposing it. You’ll be able to track progress on this topic throughout the year via the Interop 2024 dashboard.

@cookiecrook
Copy link

FYI some subtests in the tracked Focus Area files will be removed due to this bug:

@zcorpan
Copy link
Member Author

zcorpan commented Mar 6, 2024

@nmlapre
Copy link

nmlapre commented Mar 11, 2024

can you file a test change proposal?

Sure thing - filed here: #645

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
focus-area-proposal Focus Area Proposal
Projects
Status: Done
Development

No branches or pull requests