Accessibility Interop Project Overview and Contribution Guidelines
The Web Platform Tests (WPT) project is a “cross-browser test suite for the Web-platform stack”. Its main goals are described on the Web Platform Tests website:
“Writing tests in a way that allows them to be run in all browsers gives browser projects confidence that they are shipping software which is compatible with other implementations, and that later implementations will be compatible with their implementations. This in turn gives Web authors/developers confidence that they can actually rely on the Web platform to deliver on the promise of working across browsers and devices without needing extra layers of abstraction to paper over the gaps left by specification editors and implementers.”
Additional resources to learn more about WPT:
- web-platform-tests GitHub repo README
- Web Platform Tests homepage
- Introduction to the web-platform-tests (YouTube)
The Interop Project is an integral component of WPT:
“The goal of the Interop Project is to improve the web by making it easier to make websites and web apps that work in every browser or browser engine at the same time.”
The Interop Project investigates specific focus areas (2024 Interop focus areas dashboard) and specifically, the Accessibility Interop group focuses on web platform accessibility testing.
From the accessibility-interop GitHub repo README:
“Contributors from Apple, Mozilla, Igalia, Bocoup, Adobe, Hilton, Microsoft, Google, and many more individual web developers collaborated to kickstart a continuous integration, interoperable, accessibility testing project in all major browsers. To date, this is the only known project that stands to improve the state of accessibility across the entire Web, regardless of which platform, browser, or assistive technology you may be using or testing.”
The group usually meets on the first Tuesday of the month to discuss current accessibility investigation and focus area topics, accessibility tests, yearly investigation progress and open issues.
To participate in the monthly Zoom meetings, please reach out to Simon Pieters (@zcorpan).
Related group: Accessible Rich Internet Applications (ARIA) Working Group.
Prior to WPT, browser vendors primarily wrote engine-specific, internal tests to check accessibility. This sometimes resulted in inconsistent behavior between browsers, whether due to testing of different edge cases or differing interpretations of the specifications. In addition, for web developers that may be frustrated by code that should “just work”, writing engine-specific tests is an opaque, non-trivial task. For example, if an ARIA attribute is supported in all but one browser, what can a developer do to get this fixed at the browser level?
Using the WPT framework, the Interop Accessibility project enables developers to write a single automated test that runs in all major browser engines; once submitted to WPT, browser bugs can then be raised, systematically tracked and ultimately resolved in a transparent manner.
By empowering web developers to make the web more interoperable, we achieve numerous accessibility benefits:
- Helping to build a web that accessible by default
- Surfacing browser implementation gaps and unexpected accessibility behaviors per specifications
- Preventing regression of accessibility bugs
- Freeing up time spent on debugging/addressing accessibility during development
To learn more, see WebKit blog: Improving Web Accessibility with Web Platform Tests.
At a high level, platform accessibility APIs expose browser-generated accessibility information about user interfaces (UI). This metadata (e.g., role, name, state) is then relayed to users of assistive technologies to understand the purpose and structure of UI:
Source: https://github.com/w3c/aria/blob/main/documentation/tests.md
Browser engines have internal, engine-specific tests however, WPT facilitates cross-browser testing via WebDriver.
We’re able to write JavaScript-based tests to test accessibility-focused web platform features using the WPT infrastructure which includes:
- testharness.js - WPT library/framework for writing tests.
- testdriver.js - WPT library for test automation (e.g., simulating a keypress).
- WebDriver - Specification that codifies platform-independent browser remote control and in particular enables automated testing. WebDriver implementations across browsers:
At present, WebDriver supports the following accessors for querying accessibility object properties:
- Get Computed Label - Returns an element’s accessible name.
- Get Computed Role - Returns an element’s WAI-ARIA role.
Note: See also web-platform-tests.org for additional function documentation on get_computed_label() and get_computed_role().
As part of the 2024 accessibility interop investigation, additional accessibility accessors are being explored to query more than label/role for an accessible node.
WPT contains tests for many web features/technologies and there is no single “accessibility” directory. Below are common locations for accessibility-focused tests:
| WPT directory (/wpt) | Tested Specifications |
|---|---|
| /accessibility | Primarily accessibility crash tests (and not for testing accessibility specs) |
| /accname | Accessible Name and Description Computation |
| /core-aam | Core Accessibility API Mappings |
| /css | For accessibility testing of CSS browser behavior. Currently, we only have CSS tests in wpt/css/css-display/accessibility. |
| /dpub-aam | Digital Publishing Accessibility API Mappings |
| /html-aam | HTML Accessibility API Mappings |
| /svg-aam | SVG Accessibility API Mappings |
| /wai-aria | ARIA, includes ARIA roles/states/properties. Here, you’ll also find useful ARIA utilities when writing tests (see wpt/wai-aria/scripts). |
See machine setup steps for Running WPT Tests for ARIA and Related Specs.
Accessibility tests are simply HTML files with some data attributes that provide hooks for WPT test execution and reporting.
As an example, here’s an excerpt from the button roles test (wpt/wai-aria/role/button-role.html):
<!doctype html>
<html>
<head>
<title>Button-related Role Verification Tests</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<script src="/resources/testdriver.js"></script>
<script src="/resources/testdriver-vendor.js"></script>
<script src="/resources/testdriver-actions.js"></script>
<script src="/wai-aria/scripts/aria-utils.js"></script>
</head>
<body>
<p>Tests <a href="https://w3c.github.io/aria/#button">button</a> and related roles.</p>
<div role="button" aria-haspopup="false" data-testname="button aria-haspopup false" data-expectedrole="button" class="ex"></div>
…
<script>
AriaUtils.verifyRolesBySelector(".ex");
</script>
</body>
</html>Notable callouts:
- The
<title>tag serves as the page title. Usually, this is the name of the test, e.g., “Button-related Role Verification Tests”. - The
<script>tags in the import external WPT scripts required for test execution. - The actual test declaration, e.g., in the example above, we have a
<div role=”button”>that we’re asserting has a “button” role. Specifically, note thedata-expectedroleattribute used to assert what we’re looking for, and thedata-testnameattribute that serves as a test descriptor. - The
<script>block at the bottom of the<body>tag references a helpful function. Here, we’re using the ARIA utilityverifyRolesBySelector()function on all elements matching the.exselector to verify their role.
Returning to our button example above, let’s look at this test:
<div role="button" aria-haspopup="false" data-testname="button aria-haspopup false" data-expectedrole="button" class="ex"></div>It would be cumbersome to write WebDriver accessibility accessor functions for every HTML test file. To simplify testing efforts, what we want to do is write HTML markup, provide a test name (via data-testname) and assert some accessibility information about the DOM element (e.g., WebDriver get_computed_role() returns "button" for this example test).
Fortunately, we have ARIA helper functions available in wpt/wai-aria/scripts/aria-utils.js. Our button test uses verifyRolesBySelector() from aria-utils.js:
/*
Tests computed ROLE of all elements matching selector
against the string value of their data-expectedrole attribute.
*/
verifyRolesBySelector: function(selector, roleTestNamePrefix) {
const els = document.querySelectorAll(selector);
if (!els.length) {
throw `Selector passed in verifyRolesBySelector("${selector}") should match at least one element.`;
}
for (const el of els) {
let role = el.getAttribute("data-expectedrole");
let testName = el.getAttribute("data-testname") || role; // data-testname optional if role is unique per test file
if (typeof roleTestNamePrefix !== "undefined") {
testName = roleTestNamePrefix + testName;
}
promise_test(async t => {
const expectedRole = el.getAttribute("data-expectedrole");
const computedRole = await test_driver.get_computed_role(el);
assert_equals(computedRole, expectedRole, el.outerHTML);
}, `${testName}`);
}
}Notable callouts:
- We don’t need to call the WebDriver accessibility accessors (i.e.,
get_computed_role(),get_computed_label()) in this test because it happens withinverifyRolesBySelector()here:
const computedRole = await test_driver.get_computed_role(el);- For each of our tests, the ARIA utility functions use a Promise test which is a method of reliably executing code in an asynchronous manner. The important thing to note here is the assertion:
assert_equals(computedRole, expectedRole, el.outerHTML);Note: This section is mostly informative. Besides including aria-utils.js in your test file’s <script> block, you won’t be interacting much with aria-utils.js unless new global functionality is needed to support test authoring.
After machine setup (see “Getting Started” above), the general process for writing accessibility tests:
- Create a new Interop Accessibility issue, or assign yourself an existing issue (e.g., from ARIA, Accessibility Interop repos). For existing issues, we suggest starting with issues tagged “good first issue”
- Determine where the accessibility test file should reside (see “WPT Accessibility Directories” section)
- After writing the test, submit a pull request (PR)
- Address feedback from others and iterate on the test
- Once approved, merge PR
Below is example HTML scaffolding that can be used as a template for new test files:
<!doctype html>
<html>
<head>
<title>MY TEST NAME</title>
<script src="/resources/testharness.js"></script>
<script src="/resources/testharnessreport.js"></script>
<script src="/resources/testdriver.js"></script>
<script src="/resources/testdriver-vendor.js"></script>
<script src="/resources/testdriver-actions.js"></script>
<script src="/wai-aria/scripts/aria-utils.js"></script>
</head>
<body>
<h1>MY TEST NAME</h1>
<p>Tests <a href="W3C-Spec-Link">W3C SPEC HERE</a> AND SOME ADDITIONAL CONTEXT HERE.</p>
<!--
YOUR TESTS GO HERE
-->
<script>
/*
ARIA Utils you may need which is usually one or more of the following:
AriaUtils.verifyLabelsBySelector(".ex-label");
AriaUtils.verifyRolesBySelector(".ex-role");
AriaUtils.verifyGenericRolesBySelector(".ex-generic");
*/
</script>
</body>
</html>Some helpful tips to streamline test authoring and ensure test consistency across accessibility interoperability work:
- Validate HTML such as with the W3C’s Markup Validation Service
- Always run tests locally (see “Test Execution” section)
- Provide links to relevant parts of a specification or standard being tested
- Provide comments for related/open issues as needed, e.g.,
<!-- Embedded controls tested in ./comp_embedded_control.html -->- Name your HTML file descriptively, e.g.,
aria-hidden-tested-via-label.html - Alphabetize tests where possible, and/or provide structure via heading tags (e.g.,
<h2>) to better organize tests - Ensure that test names supplied via
data-testnameare globally unique within the WPT Project. Generally, more descriptive (and verbose) test names are preferred - Ensure
ids when referencing elements are unique (e.g.,<span id=”s7”>) - Links without a destination can use an
hrefvalue of “#”, e.g.,<a href=”#”> - Images without a URL source can use a
srcvalue ofsrc="data:image/gif;base64,R0lGODlhAQABAIAAAP///wAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw== - For PRs
- PRs should use descriptive names, e.g., “display: contents accessibility tests”
- Ensure all PR checks have passed which includes a lint check, e.g.,:
It’s strongly encouraged to take a look at past accessibility tests (see “WPT Accessibility Directories” section) to familiarize yourself with existing tests, testing approaches, test file structure and general inspiration!
WPT supports a robust continuous integration (CI) environment maintained by W3 members. This means that tests pushed to the WPT repo are automatically run and published to the live WPT dashboard so results can be seen in real-time across the latest, publicly released browser engines.
Please see below for a high-level overview of WPT and how tests interact with the WPT infrastructure:
Tests are executed in two ways:
Tests are executed in your terminal using the following wpt command:
./wpt run --log-mach-level debug --log-mach - --webdriver-arg="--diagnose" <BROWSER> <FILE PATH>For example, the following browser-specific commands run the existing name calculation test (https://github.com/web-platform-tests/wpt/blob/master/accname/name/comp_label.html); note the parameters at the end of the command specifying the browser engine to use for execution and test file path:
Safari:
./wpt run --log-mach-level debug --log-mach - --webdriver-arg="--diagnose" safari /Desktop/wpt/accname/name/comp_label.htmlChrome:
./wpt run --log-mach-level debug --log-mach - --webdriver-arg="--diagnose" chrome /Desktop/wpt/accname/name/comp_label.htmlFirefox:
./wpt run --log-mach-level debug --log-mach - --webdriver-arg="--diagnose" firefox /Desktop/wpt/accname/name/comp_label.htmlNote: As part of executing WPT tests locally, depending on the browser, you may be prompted within the CLI to download the associated webdriver.
Push a commit to your test branch and view the test results using the PR checks (such as the pull request’s “Checks” tab under “wpt.fyi” checks). Select "Latest Results" to see the pass/fail stats for all engines, or "View Comparison" to see before and after results for a single engine.
If you see generic failures across the board such as this:
Try enabling the "Show Details" switch for more information... It may be an authoring error in the test file.
The following are common error messages when running tests locally:
-
“Duplicate test names” - Ensure every
data-testnameattribute has a globally unique description of the test. -
“Selector passed in
verify…()should match at least one element” - This error will occur if you use an invalid or non-existent class name in your test. Verify that the spelling of the class in your test matches the spelling of the selector in your JavaScript. -
“Action get_computed_label/get_computed_role failed” - This is usually a result of malformed HTML. Ensure that the HTML markup is valid, well-formed, elements are nested appropriately and start/end tags are used (where required). Performing HTML validation is strongly recommended.
-
“Expected (object) null but got…” - This error can occur if the test contains malformed data attributes. Ensure that all
aria-utils.jsdata-*attributes are expected values and spelled correctly, namely:data-testname,data-expectedlabel,data-expectedrole.
A: Accessibility interop work relies on contributions from individuals and browser vendors. Your support is needed and welcome!
From the Accessibility Interop repo:
If you're interested in contributing or learning more, start by reviewing the open issues list, or reviewing an open pull request.
A: WPT uses WebDriver for querying accessibility information about DOM elements. At present, this is limited to an element’s label or role (see “WebDriver & Accessibility Introspection” section).
A: This is being explored as a priority for 2024 accessibility interop work. Discussion on this can be found in AOM #203.
A: Please check with Accessibility Interop members (mainly James Craig (@cookiecrook) to ensure it’s appropriately located.
Q: Do I need to run tests locally since tests pushed to the WPT repo are executed automatically and results subsequently published to the WPT reporting dashboard (wpt.fyi)?
A: Yes, always run tests locally for several reasons:
- it reduces CI resource usage
- allows you to see diffs between local and CI execution results
- allows instant test execution without pushing commits
A: See “Writing WPT Accessibility Tests - Troubleshooting” section.
Q: I need to test something that isn’t available via aria-utils.js. What should I do?
A: There may be instances where a test requires additional test-specific logic unavailable in aria-utils.js (e.g., a function that walks a DOM tree and verifies a property for an element’s ancestors). As done across other WPT work, this extra functionality can be added directly in the file’s <script>. If you think the functionality should be made available as a global utility, please raise an issue or reach out to an Accessibility Interop member.
Q: Accessibility interop work seems to focus on assistive technology accessibility. What about testing other aspects of web accessibility, e.g., the visual or keyboard experience?
A: The group is currently investigating keyboard accessibility testing as part of interop-accessibility #101. If you have additional ideas to expand testing coverage, please let us know!
A: See web-platform-test.org’s testharness.js page on “Assert” functions.