Addon-Test: Implement Addon Test TestProvider Backend

[valentinpalkovic]

Closes #29095

What I did

This PR introduces changes aimed at improving Storybook's Vitest integration by enabling the user to trigger a test run from Storybook's UI. The key changes include the addition of Vitest runner management functionality, integration with Storybook’s UI, and dependency updates to support improved functionality.

Vitest Configuration Updates

Modified Storybook UI's vitest.config.ts to include all stories (core, addons, lib) for testing. This was done temporarily to trigger tests for all stories, ensuring comprehensive testing of all components. This change has to be reverted because, currently, not all stories are running successfully in a Vitest Test environment.

Dependency Updates

Upgraded @vitest/browser and vitest from ^2.0.0 to ^2.1.1 to leverage newer features and stability improvements. We have to make sure, though, to remain compatible with < 2.1.1 versions. Unfortunately, a lot of types and internal API's of Vitest have changed significantly between 2.0 and 2.1.

Introduction of boot-test-runner.ts

A new file, boot-test-runner.ts, was added to manage child processes responsible for running Vitest tests. It restarts the Vitest runner upon failure, ensuring reliability. This process monitors Storybook core events such as TESTING_MODULE_RUN_REQUEST and TESTING_MODULE_CANCEL_TEST_RUN_REQUEST for running or canceling tests.

Custom Test Reporter (StorybookReporter)

Introduced StorybookReporter, a custom Vitest reporter that integrates Vitest's test reporting with Storybook's core events. It captures test progress and emits detailed results, including test assertions and failures, which are communicated to Storybook via the established channel.

Test Manager (test-manager.ts)

A new module to handle the orchestration of Vitest runs. It manages the Vitest lifecycle, including starting and stopping the test process, handling requests from the Storybook UI, and emitting progress updates to the UI.

Vitest Manager (vitest-manager.ts)

The Vitest Manager is a key component introduced to handle the orchestration and lifecycle of Vitest processes. It ensures proper initialization, execution, and termination of test runs, both for individual test cases and all tests at once.

The Vitest Manager communicates with the Test manager. It also ensures that Storybook test-specific settings, such as "watch mode," are honored during test runs.

The manager dynamically identifies the tests to run, either based on Storybook story imports or requests from the UI, ensuring that only the relevant stories are tested.

Sidebar Integration

Integrated the Vitest test runner into the Storybook sidebar. A new "Run test for all Stories" button was added, allowing users to trigger tests directly from the Storybook UI. A play button for the root Story groups was added as well to run tests for a specific set of stories. The sidebar now also includes a checkbox to toggle "Watch Mode," enabling continuous test watching for real-time feedback. All this change was introduced ONLY for testing purposes and doesn't fulfill or has anything to do with a proper solution, which implements event emitting/listening in a more generic fashion.

New Constants and Events

Added new core events (TESTING_MODULE_RUN_REQUEST, TESTING_MODULE_CANCEL_TEST_RUN_REQUEST, TESTING_MODULE_RUN_ALL_REQUEST, and more) to facilitate communication between the Storybook UI and the test runner.

Checklist for Contributors

Testing

The changes in this PR are covered in the following automated tests:

• stories
• unit tests
• integration tests
• end-to-end tests

Manual testing

• cd ./code
• yarn
• yarn nx run-many -t build --parallel=9
• yarn storybook:ui
• Visit Storybook and open your developer console
• For testing purposes, new UI elements were added to the sidebar to trigger test runs. The play button for the story groups triggers a test run for all containing stories. The "Run test for all stories" button triggers a test run for all stories and the watch mode checkbox triggers "watch mode" requests for the backend. When watchmode is triggered, tests are not automatically running. A new test run has to be triggered.

• Go to the "Network" tab and take a look at the Websocket Events. Notice how the proper events are sent and are received.

Documentation

• Add or update documentation reflecting your changes
• If you are deprecating/removing a feature, make sure to update
  MIGRATION.MD

Checklist for Maintainers

• When this PR is ready for testing, make sure to add ci:normal, ci:merged or ci:daily GH label to it to run a specific set of sandboxes. The particular set of sandboxes can be found in code/lib/cli/src/sandbox-templates.ts

• Make sure this PR contains one of the labels below:

  Available labels

  • bug: Internal changes that fixes incorrect behavior.
  • maintenance: User-facing maintenance tasks.
  • dependencies: Upgrading (sometimes downgrading) dependencies.
  • build: Internal-facing build tooling & test updates. Will not show up in release changelog.
  • cleanup: Minor cleanup style change. Will not show up in release changelog.
  • documentation: Documentation only changes. Will not show up in release changelog.
  • feature request: Introducing a new feature.
  • BREAKING CHANGE: Changes that break compatibility in some way with current major version.
  • other: Changes that don't fit in the above categories.

🦋 Canary release

This PR does not have a canary release associated. You can request a canary release of this pull request by mentioning the @storybookjs/core team here.

core team members can create a canary release here or locally with gh workflow run --repo storybookjs/storybook canary-release-pr.yml --field pr=<PR_NUMBER>

name	before	after	diff	z	%
createSize	0 B	0 B	0 B	-	-
generateSize	0 B	77.5 MB	77.5 MB	-	100%
initSize	0 B	163 MB	163 MB	-	100%
diffSize	0 B	85 MB	85 MB	-	100%
buildSize	0 B	7.58 MB	7.58 MB	-	100%
buildSbAddonsSize	0 B	1.66 MB	1.66 MB	-	100%
buildSbCommonSize	0 B	195 kB	195 kB	-	100%
buildSbManagerSize	0 B	2.35 MB	2.35 MB	-	100%
buildSbPreviewSize	0 B	353 kB	353 kB	-	100%
buildStaticSize	0 B	0 B	0 B	-	-
buildPrebuildSize	0 B	4.56 MB	4.56 MB	-	100%
buildPreviewSize	0 B	3.02 MB	3.02 MB	-	100%
testBuildSize	0 B	0 B	0 B	-	-
testBuildSbAddonsSize	0 B	0 B	0 B	-	-
testBuildSbCommonSize	0 B	0 B	0 B	-	-
testBuildSbManagerSize	0 B	0 B	0 B	-	-
testBuildSbPreviewSize	0 B	0 B	0 B	-	-
testBuildStaticSize	0 B	0 B	0 B	-	-
testBuildPrebuildSize	0 B	0 B	0 B	-	-
testBuildPreviewSize	0 B	0 B	0 B	-	-

name	before	after	diff	z	%
createTime	0ms	18.9s	18.9s	-	100%
generateTime	0ms	23.5s	23.5s	-	100%
initTime	0ms	17.5s	17.5s	-	100%
buildTime	0ms	10.7s	10.7s	-	100%
testBuildTime	0ms	0ms	0ms	-	-
devPreviewResponsive	0ms	7.2s	7.2s	-	100%
devManagerResponsive	0ms	4.7s	4.7s	-	100%
devManagerHeaderVisible	0ms	927ms	927ms	-	100%
devManagerIndexVisible	0ms	973ms	973ms	-	100%
devStoryVisibleUncached	0ms	1.5s	1.5s	-	100%
devStoryVisible	0ms	970ms	970ms	-	100%
devAutodocsVisible	0ms	842ms	842ms	-	100%
devMDXVisible	0ms	771ms	771ms	-	100%
buildManagerHeaderVisible	0ms	773ms	773ms	-	100%
buildManagerIndexVisible	0ms	774ms	774ms	-	100%
buildStoryVisible	0ms	875ms	875ms	-	100%
buildAutodocsVisible	0ms	707ms	707ms	-	100%
buildMDXVisible	0ms	798ms	798ms	-	100%

[nx-cloud]

☁️ Nx Cloud Report

CI is running/has finished running commands for commit f8095ba. As they complete they will appear below. Click to see the status, the terminal output, and the build insights.

📂 See all runs for this CI Pipeline Execution

---

✅ Successfully ran 1 target

• nx run-many -t build --parallel=3

---

Sent with 💌 from NxCloud.

[greptile-apps]

_{15 file(s) reviewed, 12 comment(s)
Edit PR Review Bot Settings}