[Tracking]: Unified a11y tests

[valentinpalkovic]

🧑‍🤝‍🧑 Who: @kasperpeulen and @valentinpalkovic

This is a tracking issue for the Unified a11y tests project. Its purpose is to keep track of the overall status of the project and tasks and plan everything around it.

⚠️ Problem

Accessibility (a11y) is increasingly required in modern UIs. This is already enforced for many government services around the world, and is also becoming mandated for private services as well, e.g. European accessibility act which goes into law in June 2025.

Today, Storybook’s accessibility addon addon-a11y is one of the most popular addons not bundled with Storybook at 1.6M weekly npm dl’s. It provides a way to interactively test and debug a11y violations in stories

The biggest a11y product gap today is that you have to manually click on each story to see the a11y violations for that specific story. There is no way to do this across all stories in batch.

If we can solve this problem in the context of our Unified UI Testing project, it will go a long way towards improving support for A11y workflows in Storybook AND make our unified testing UI that much more compelling.

🏁 Goals

1. As a Storybook user, I want to be able to run a11y tests across all the stories in my Storybook, so that I can see the full a11y status of my project.

🚩 Milestones

Finalize API's - Deadline: 13th Nov

Preview Give feedback

1. Finalize Reporting API specification - Event name and payload conventions | capabilities
2. Finalize afterEach specification
3. Investigate discrepancy cases and how to deal with them

Technical Preparations - Deadline: 22th Nov

Preview Give feedback

1. Core: Implement afterEach #29584
   ci:normal feature request +2
   
2. Core: Introduce Reporting API and STORY_COMPLETED Event #29608
   ci:normal feature request +2
   

A11y Implementations - Deadline: 6th Dec

Preview Give feedback

1. Register a11y Sub Test Provider (Batch run and single story run)
2. Change a11y reporting mechanism - Use Reporting API
3. Implement Vitest report proxy of a11y test results
4. Support new parameters and globals for a11y #29682
   accessibility addon: a11y ci:normal feature request needs qa +5
   
5. Add discrepancy handling to A11yPanel #29661
   accessibility ci:normal feature request needs qa +4
   
6. [Scope increase]: Support a11y tags to enable/disable automatic a11y runs
7. [Scope increase]: Automigration: Automatically add addon-a11y to .storybook/vitest.setup.ts when upgrading to > 8.5.0
8. [Scope increase]: Addon-Test postinstall script: Consider addon-a11y if it is already installed
9. Documentation
10. Refactor status handling in frontend (Remove https://github.com/storybookjs/storybook/pull/29643/files#diff-0912d15d65479bd34f1802061483276d04484fdf7359fd9f9a53b1decd323b8eR147-R186 and sync with Gert)
11. Adjust version range for automigration (https://github.com/storybookjs/storybook/pull/29643/files#diff-f0bd0ad5176683fc69b35a8f3d857e7e901b8a1d439d6aa65dad42dd8efbb071R25-R26)
12. Merge Addon A11y: Create a11y test provider and revamp a11y addon #29643

After QA

Preview Give feedback

1. Blog post. “Component testing for Accessibility” highlighting the new test behavior, filtering, and debugging workflow. #29644
   +0
   
2. Release @storybook/csf and update its version in monorepo
3. Update "init" components to be accessible for each framework
4. Adjust CI to adjust sandbox generation to include addon-a11y + writing E2E tests

Bugs

Preview Give feedback

1. @storybook/addon-a11y/preview -> Typescript reports error in certain configurations
2. The a11y addon panel's tab doesn't show the violation number anymore
3. preview.js file is cached aggressively by vitest - restart of Storybook necessary
4. In Vitest 2.1.4 and later, the status of Accessibility in the Testing module remains unknown

Nice to have

Preview Give feedback

1. Add postinstall script to @storybook/addon-a11y to update vitest.setup.<ts|js> file if @storybook/experimental-addon-test is installed
2. A11y addon improvements. Rework addon experience per Accessibility addon V2 design
3. Investigate: Optimize accessibility testing to manage inherited violations in Storybook components
4. fix(addon-a11y): Addon-a11y displays visual bug when running the tests #29552
   addon: a11y bug ci:normal needs qa +4
   
5. Improve accessibility of our own Storybook components

🚩 Out of scope

• Create a unified API to replace Status API, Testing Module API and Reporting API.