Core: Add unified UI Testing Module

[yannbf]

Closes #29088

This PR introduces a new UI element to the Storybook sidebar called the "Testing Module", and provides a set of APIs which allow addons to register themselves as a "test provider".

The testing module lists all registered test providers and provides the capability to trigger a test run or toggle watch mode for each test provider, depending on what the provider supports.

Here's a basic example for how to register a test provider:

addons.register(ADDON_ID, (api) => {
  addons.add(`${ADDON_ID}/test-provider`, {
    type: Addon_TypesEnum.experimental_TEST_PROVIDER,
    runnable: true,
    watchable: true,
    title: ({ crashed }) => (crashed ? "Tests didn't complete" : 'Component tests'),
    description: ({ failed, running, watching, progress, crashed, details }) => {
      if (failed) return "Failed";
      if (running) return "Running";
      return "Ready";
    },
    mapStatusUpdate: ({ details }) => {
      // Return a Record<StoryId, API_StatusObject> here 
      // to set story statuses based on test results
    }
  });
});

Test providers can listen for these events on the channel:

• TESTING_MODULE_RUN_REQUEST
• TESTING_MODULE_RUN_ALL_REQUEST
• TESTING_MODULE_CANCEL_TEST_RUN_REQUEST
• TESTING_MODULE_WATCH_MODE_REQUEST

Test providers should emit these events to update the Testing Module state:

• TESTING_MODULE_CRASH_REPORT
• TESTING_MODULE_PROGRESS_REPORT
• TESTING_MODULE_CANCEL_TEST_RUN_RESPONSE

For a complete example, check out the Component Tests addon.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

• stories
• unit tests
• integration tests
• end-to-end tests

Manual testing

Projects to test:

• Storybook
• Chromatic
• Mealdrop
• Pseudo States Addon (older Storybook with new VTA)

For each project, make sure you are using the right dependency versions:

• Storybook: Canary version specified below (except for Pseudo States addon which uses 8.2.9)
• Visual Tests addon: Canary version specified here

To QA:

• Storybook add command:
  • npx storybook@<version> add @storybook/experimental-addon-test@<version>
• Testing Module
  • Does it look good, resize properly, is it hidden on mobile?
  • Is it collapsible and does it not overlap the sidebar tree view?
  • Does it run tests for a single test provider (using their play button)?
  • Does it run tests for all test providers (using the Run tests button)?
• Component tests
  • Can you run tests?
  • Can you cancel a test run?
  • Can you toggle watch mode and does it run tests on file save?
  • Does it report status changes in the testing module?
  • Are buttons disabled when they should be?
  • Does it update the sidebar with status dots (on warning/error)?
  • Does it properly update the warnings/errors count filter buttons?
• Visual tests
  • Can you trigger a test run?
  • Can you cancel a test run (before upload is finished)?
  • Does it report status changes in the testing module?
  • Are buttons disabled when they should be?
  • Does it update the sidebar with status dots (on warning/error)?
  • Does it properly update the warnings/errors count filter buttons?
• Old VTA in new Storybook
  • Does it render the old Play button at the top?
  • Do visual changes show up as warnings in the testing module?

Documentation

• Add or update documentation reflecting your changes
• If you are deprecating/removing a feature, make sure to update
  MIGRATION.MD

Checklist for Maintainers

• When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts

• Make sure this PR contains one of the labels below:

  Available labels

  • bug: Internal changes that fixes incorrect behavior.
  • maintenance: User-facing maintenance tasks.
  • dependencies: Upgrading (sometimes downgrading) dependencies.
  • build: Internal-facing build tooling & test updates. Will not show up in release changelog.
  • cleanup: Minor cleanup style change. Will not show up in release changelog.
  • documentation: Documentation only changes. Will not show up in release changelog.
  • feature request: Introducing a new feature.
  • BREAKING CHANGE: Changes that break compatibility in some way with current major version.
  • other: Changes that don't fit in the above categories.

🦋 Canary release

This pull request has been released as version 0.0.0-pr-29241-sha-f30996d6. Try it out in a new sandbox by running npx [email protected] sandbox or in an existing project with npx [email protected] upgrade.

More information

Published version	0.0.0-pr-29241-sha-f30996d6
Triggered by	@valentinpalkovic
Repository	storybookjs/storybook
Branch	unified-ui-testing
Commit	f30996d6
Datetime	Wed Oct 23 13:01:14 UTC 2024 (1729688474)
Workflow run	11480425464

To request a new release of this pull request, mention the @storybookjs/core team.

core team members can create a new canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=29241

name	before	after	diff	z	%
createSize	0 B	0 B	0 B	-	-
generateSize	78.7 MB	77.7 MB	-1.04 MB	-90.67	-1.3%
initSize	147 MB	146 MB	-898 kB	-23.44	-0.6%
diffSize	68.4 MB	68.5 MB	139 kB	9.52	0.2%
buildSize	6.8 MB	7.1 MB	304 kB	205.12	4.3%
buildSbAddonsSize	1.5 MB	1.79 MB	284 kB	22564.44	🔺15.9%
buildSbCommonSize	195 kB	195 kB	0 B	-	0%
buildSbManagerSize	1.84 MB	1.85 MB	15.7 kB	12.59	0.8%
buildSbPreviewSize	270 kB	271 kB	1.13 kB	432.6	0.4%
buildStaticSize	0 B	0 B	0 B	-	-
buildPrebuildSize	3.8 MB	4.1 MB	301 kB	203.17	🔺7.3%
buildPreviewSize	2.99 MB	3 MB	2.87 kB	1095.61	0.1%
testBuildSize	0 B	0 B	0 B	-	-
testBuildSbAddonsSize	0 B	0 B	0 B	-	-
testBuildSbCommonSize	0 B	0 B	0 B	-	-
testBuildSbManagerSize	0 B	0 B	0 B	-	-
testBuildSbPreviewSize	0 B	0 B	0 B	-	-
testBuildStaticSize	0 B	0 B	0 B	-	-
testBuildPrebuildSize	0 B	0 B	0 B	-	-
testBuildPreviewSize	0 B	0 B	0 B	-	-

name	before	after	diff	z	%
createTime	10.3s	17.4s	7.1s	0.8	40.9%
generateTime	24.6s	22.9s	-1s -695ms	0.09	-7.4%
initTime	17s	15.8s	-1s -180ms	0.1	-7.4%
buildTime	9.5s	8.5s	-971ms	-0.31	-11.4%
testBuildTime	0ms	0ms	0ms	-	-
devPreviewResponsive	6.1s	8.2s	2s	2.38	🔺25.4%
devManagerResponsive	4.2s	5.5s	1.3s	2.9	🔺23.6%
devManagerHeaderVisible	521ms	779ms	258ms	2.14	🔺33.1%
devManagerIndexVisible	557ms	821ms	264ms	2.13	🔺32.2%
devStoryVisibleUncached	845ms	1.4s	645ms	2.26	🔺43.3%
devStoryVisible	555ms	818ms	263ms	2.09	🔺32.2%
devAutodocsVisible	491ms	661ms	170ms	1.66	🔺25.7%
devMDXVisible	493ms	659ms	166ms	2.2	🔺25.2%
buildManagerHeaderVisible	601ms	711ms	110ms	1.82	🔺15.5%
buildManagerIndexVisible	626ms	725ms	99ms	1.52	🔺13.7%
buildStoryVisible	627ms	709ms	82ms	1.24	🔺11.6%
buildAutodocsVisible	442ms	579ms	137ms	0.81	23.7%
buildMDXVisible	455ms	626ms	171ms	1.76	🔺27.3%

Greptile Summary

This PR introduces a unified UI testing module to Storybook, focusing on integrating Vitest for component testing. Here are the key changes:

• Added new @storybook/experimental-addon-test package with Vitest integration
• Implemented TestManager and VitestManager classes for managing test execution
• Created a new sidebar component for running tests and toggling watch mode
• Added event handling for test-related actions in the Storybook core
• Introduced new types and interfaces for test providers and results
• Updated the Storybook configuration to support the new testing features

Key points to note:

• The implementation is marked as experimental, indicating potential future changes
• New test runner boots on-demand to optimize performance
• Added support for running tests on individual stories and all stories
• Implemented progress reporting and status updates for test runs
• Some TODOs and potential improvements are noted in the code comments