RFC 204: WPT testing for AAMs (platform accessibility APIs)

[spectranaut]

Here is a link to the markdown file rendered: https://github.com/Igalia/rfcs/blob/wpt-for-aams-rfc/rfcs/wpt-for-aams.md

There is a lot of background context in the RFC for those who wish to understand more about the underlying accessibility technologies.

Here are the notes for the WPT meeting where this was first brought up: https://pad.0b101010.services/wpt-infra-meeting?view

[annevk]

Per-platform tests

I think it would be ideal if we could avoid per-platform tests by instead expressing the per-platform mapping for a given feature in some configuration file that's included in tests. I think that will make it easier to contribute new tests as you don't necessarily have to know about the platform mappings (unless you're adding coverage for a completely new feature) and increase the likelihood of cross-platform coverage (platforms will only have to update these configuration files and not duplicate tests and such).

[zcorpan]

@annevk interesting idea. The tests would assert that the appropriate mapping is used (e.g., core-aam "aria-autocomplete=inline, list, or both" or html-aam "input (type attribute in the Color state)").

[spectranaut]

@annevk I remember talking with you about this at the webengine hacktest -- I think this might be a good idea and I've started to explore it. In this scenario, this json file contains mappings defined in a single table, like @zcorpan mentioned -- core-aam's aria-autocomplete=inline, list, or both and html-aams input (type attribute in the Color state). Then the test will check the mapping appropriate for that platform as defined in the JSON.

The core-aam manual tests already have this data in a similar structure, see the manual test for alert which has a javascript object with the mapping expectations for each platform: https://github.com/web-platform-tests/wpt/blob/master/core-aam/alert-manual.html#L18

[spectranaut]

Ok, @annevk, here are some tests designed using that use a configuration file containing all of the platform mapping details: Igalia/wpt#14

We can write tests that do not look operating system dependent this way, but, the results of the test are operating system dependent. I tried to get around this, but its a bit awkward -- in this design, every test expands to 4 subtests, one per API, regardless of the operating system the test was run on.

Ideally, we would have a single subtest that says to the backend: "Hey, test the appropriate API and let us know the result", but we can't because of operating systems and APIs are not a 1-1 mapping. Windows has two accessibility APIs (UIA and IAccessible2) and we need to know the status of both APIs separately (different ATs use different APIs, both need to be supported). So at minimum, we need different 2 subtest for windows, where as we only need one for linux and one for macOS.

I wrote about some alternatives test designs in the link PR's description. They could probably all be used with this "configuration file" idea, the configuration file design doesn't solve the platform specific test and/or results problem.

[annevk]

That is tricky! Would it be possible that for these tests we handle the operating system and API together as a single platform? So we run the test for Windows UIA and again for Windows IAccessible2? (Likely with skips when we already know the combination isn't supported for a given browser.)

This would impact how we have to display the results for these tests (and possibly how we run them, but that already needs changes anyway), but it would keep everything else consistent.

[spectranaut]

Hmm I think I'm not totally sure I understand the implications of your suggestion, or, it's what I'm already doing :) Here is an example of the linked test designs "results" for a run on linux:
https://spectranaut.github.io/examples/wpt/role_blockquote_test.html

You have to expand the asserts to understand what is really going on. Only the linux test (Atspi) ran any assertions, and passed. On Windows, both the UIA and IAccessible2 rows would have data. The APIs that don't apply to the platform just trivially pass with no assertions.

Essentially, a utility function asks the back end "can you run this test for this API" for each API, and the back end either sends a result or "no I can't run that test".

[annevk]

It's a bit hard to explain but what I'm contemplating is that you'd have a single subtest, but that e.g. Firefox Linux, Firefox macOS, Firefox Windows UIA, and Firefox Windows IAccessible2 all have their own run for that single subtest. And the same for other browsers. Skipping variants that are not applicable. And the subtest expectations would adjust based on the platform AT API that is running.

Basically I'm saying that we solve your

Ideally, we would have a single subtest that says to the backend: "Hey, test the appropriate API and let us know the result", but we can't because of operating systems and APIs are not a 1-1 mapping.

by making the APIs count and "run" these tests and not the operating systems, if you will.

[zcorpan]

Reporting results to wpt.fyi separately for each (OS, accessibility API, browser) combination allows for side-by-side comparison. However, we wouldn't want to run all of wpt for each accessibility API.

Maybe there can be one combination where all tests are run, and for the remaining applicable accessibility APIs only the AAM tests are run.

[annevk]

However, we wouldn't want to run all of wpt for each accessibility API.

Right, this is what we would have to special case for "this directory" (or some other signifier) in a couple of places.

[spectranaut]

Ah now I understand what you are trying to say. I would be nice to see the support for "blockquote" as a single row across all API/OS/Browser combinations and it seems like the right direction to go in.

We already need a special case for this directory, because these are the only tests that need "accessibility turned on". The browser typically does not compute the accessibility tree/accessibility APIs.

So in this scenario, we would need to:

• change the data collection for all the browser/OS pairs for wpt.fyi, to collect data for that yet-untested API. At this time I'm not sure where all that data collection is happening.
• update wpt.fyi to show the API information in the column for the special folders.
• update the harness to be able to specify which API to test.

[jcsteh]

I think it would be ideal if we could avoid per-platform tests by instead expressing the per-platform mapping for a given feature in some configuration file that's included in tests.

I have a couple of concerns with a declarative config style approach:

1. It separates the expected results from the tests. I'd argue tests are a lot easier to read, understand and write when there is less indirection. To be fair, the indirection is only one level deep here and I do also follow the downside of having separate tests, though I wonder whether that could be resolved by ensuring the tests for all platforms are always in the same file.
2. This works well enough when dealing with simple property comparisons. However, mappings aren't always simple properties. Examples are text attributes, table navigation and accessibility events. I'm sure we could find a way to express these declaratively, but I think it would become difficult to work with.

I think that will make it easier to contribute new tests as you don't necessarily have to know about the platform mappings

I don't quite follow this. New tests will necessarily have different expected results, which means the mapping to platforms will be different, even if that's just that there are different values for the different properties. And if you don't know the platform mappings, I don't quite follow why you'd be writing a new AAM test, since AAMs are all about platforms. I guess many non-core AAM entries map to ARIA roles/states, so those could be common somehow, but Core AAM in particular (and to a lesser extent other AAMs) inherently requires an understanding of platform mappings. I feel like I'm missing something here, though. What's the scenario where a new AAM test would be added without any concern for platform mappings?

[jcsteh]

The tests may be flaky due to timing issues with the accessibility tree being built. The accessibility tree lags behind the DOM and even paint, with no feedback available to the page about what the status of the accessibility tree is.

This could potentially be solved by waiting for events. I wonder whether this new API might need to support events to solve this problem.

[annevk]

What's the scenario where a new AAM test would be added without any concern for platform mappings?

Some possible scenarios:

1. You are contributing a new test that uses an existing mapping. E.g., role=search has coverage, but <search> did not. It would be straightforward to add coverage for <search>.
2. You are contributing a new test that does not use an existing mapping. In this case you need to know about at least one platform, but you don't need to know about all the platforms. Those can be filled in by others later.

I'm sure we could find a way to express these declaratively, but I think it would become difficult to work with.

I don't think it has to necessarily be declarative. Just abstracted.

[spectranaut]

The tests may be flaky due to timing issues with the accessibility tree being built. The accessibility tree lags behind the DOM and even paint, with no feedback available to the page about what the status of the accessibility tree is.

This could potentially be solved by waiting for events. I wonder whether this new API might need to support events to solve this problem.

Right, in many scenarios we can wait for the appropriate event on each platform before testing -- for example, before starting the test, we could listen for document:load-complete on linux, and I did have this implemented at one point in the prototype. We switched to polling for whether the tab exist with the correct URL in this scenario.

Eventually, these tests will need to support events for testing events described in the AAMs, and we will probably be able to resolve most if not all sources of flakiness by listening for events. Maybe we are being overly cautious by saying the tests might be flaky...