[Tracking]: Unified UI Testing 🔮

[ghengeveld]

🧑‍🤝‍🧑 Who: @ghengeveld, @yannbf and @valentinpalkovic

This is a tracking issue the Unified UI Testing 🔮 project. The purpose of this issue is to keep tracking of the overall status of the project and tasks, and plan everything around it.

⚠️ Problem

Storybook has the potential to be the go-to tool for comprehensive UI testing, unifying both the existing Visual Regression Testing (VRT) and Component Testing (CT) experiences. However, the current implementation falls short in two key areas:

1️⃣ Fragmented workflow

As of 8.3, users begin their Storybook CT experience with Vitest via the CLI, then transition into Storybook for visual debugging. More specifically, this process includes:

1. Configuring the storybookScript option in the Vitest plugin.
2. Running vitest in watch mode from the CLI, which spawns Storybook in the background.
3. Viewing test failure outputs in the CLI and clicking a link to open Storybook for visual debugging.

This workflow centers on Vitest, making Storybook feel like a "sidekick" (albeit a powerful one). The constant switching between Vitest and Storybook disrupts the development flow and makes the process feel fragmented.

2️⃣ Incoherent design and under-utilized CT experience

Storybook already provides a powerful VRT experience via Visual Test Addon. We have an opportunity to create a more coherent and optimized experience that fully leverages both VRT and CT capabilities within Storybook.

🏁 Goals

The high-level goals of this project are:

• Introduce a way to trigger test runs from the Storybook Manager UI.
• Support both the Storybook Test addon (Vitest) and the Visual Test addon (Chromatic).
• Support scoping down to groups, components and stories (when supported by the test provider).
• Update (aggregate) story status indicators on groups, components and stories with new icons.

🚩 Milestones

1. Testing module essentials

Beta Give feedback

1. Spec and implement Test Provider API #29092
   +0
   
2. Spec for Channel events to trigger test run #29093
   +0
   
3. Update Storybook Test addon to register with Test Provider API #29094
   +0
   
4. Update Storybook Test addon to listen for test run events #29095
   +0
   
5. Testing module UI #29096
   +0
   
6. Group/component/story dropdown menu UI update #29214
   +0
7. Bring addon-interactions functionality into addon-test #29215
   +0
   
8. E2E test #29213
   +0
   
9. Update Notifications UI to not overlap Testing Module
10. Testing Module status updates #29262
    +0
    
11. Fix cache issues with direct story test run #29289
    +0
    
12. Clear story statuses on Storybook session over
13. Uninstall addon-interactions on sb add addon-test
14. Update sandboxes to use addon-test rather than addon-interactions

2. Discrepancy handling

Beta Give feedback

1. Discrepancy detection #29162
   +0
   
2. Global error state #29312
   +0
   
3. Send interactions log from Vitest to manager #29216
   +0
4. Update test log to handle discrepancy in test results #29120
   +0
5. Additional telemetry #29349
   addon: test feature request telemetry +3
   

3. Updating VTA

Beta Give feedback

1. Update VTA to use Test Provider API #29097
   +0
2. Update VTA onboarding to point at Testing module #29098
   +0

4. Static builds

Beta Give feedback

1. Handle test results in CI #29165
   +0
2. Disable expandability of the testing module

5. Watch mode

Beta Give feedback

1. Support watchable property for test providers
2. Render an "eye" toggle button which emits a _WATCH event
3. Handle watch event in Storybook Test addon

6. Run individual stories

Beta Give feedback

1. Spec for Story Actions API #29108
   +0
2. Ellipsis story dropdown menu toggle on hover #29107
   +0
   
3. Story Actions API #29101
   +0
   
4. “Run component tests” action (Channel API)

7. Enhanced story status

Beta Give feedback

1. Add support for a score property (shown on the right in the dropdown)
2. Aggregate (sum) these counts on groups and components

8. Portable stories fixes

Beta Give feedback

1. Support new globals format for viewports in vitest plugin #29324
   +0
   
2. Add Storybook hooks support in portable stories #29574
   addon: test bug portable stories +3
3. Support renamed exports in vitest plugin e.g. export { A as B } #29244
   feature request +1
   

Out of scope

• Integrating with A11y addon
• Support for code coverage reporting

Follow-up work

• Setup a sandbox to use (and test) addon-interactions