chore: reduce flake in windows-run-app-integration-tests-chrome #29198
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…t testing certain tests that are flaky inside the pap-integration cy-in-cy pipeline. Because there are issues recording with --spec specified with cy-in-cy, we will not be recording the windows app integration tests. Additionally, this fixes the windows-test-binary-against-kitchensink-chrome where yarn is not installed on the windows-server executor by making sure yarn is installed first before any action happens when checking out a repo [run ci]
3 flaky tests on run #54656 ↗︎
Details:
|
|||||||||||||||||||||||||||||||||
| Test | Artifacts | |
|---|---|---|
| ... > can reconfigure config after CT has been set up |
Test Replay
Screenshots
|
|
cypress\e2e\scaffold-component-testing.cy.ts • 1 flaky test • launchpad-e2e
| Test | Artifacts | |
|---|---|---|
| scaffolding component testing > create-react-app > scaffolds component testing for Create React App v5 project |
Test Replay
Screenshots
|
|
cypress\e2e\migration.cy.ts • 1 flaky test • launchpad-e2e
| Test | Artifacts | |
|---|---|---|
| Full migration flow for each project > completes journey for migration-e2e-no-plugins-support-file |
Test Replay
Screenshots
|
|
Review all test suite changes for PR #29198 ↗︎
|
Took two reruns to get a passing windows pipeline, but it can pass |
cacieprins
requested changes
Mar 28, 2024
| @@ -703,6 +717,12 @@ commands: | |||
| # To run the `yarn` command, we need to walk out of the package folder. | |||
| cd ../.. | |||
|
|
|||
| if [[ "$IS_WINDOWS_APP_INTEGRATION_TEST" = true ]]; then | |||
There was a problem hiding this comment.
can we detect host os in the specs themselves, to ease tracking down why these tests are skipped in the future?
e.g.,
;(isWindowsHostCI ? describe.skip : describe)('top level describe block', function () { })
There was a problem hiding this comment.
We don't want to skip them on the OS itself though. Only in CI on a specific OS for a given test. They don't flake locally or have issues.
jennifer-shehane
approved these changes
Mar 29, 2024
cacieprins
approved these changes
Apr 2, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Additional details
Reduces flake in
windows-run-app-integration-tests-chromeby not testing certain tests that are flaky inside the pap-integration cy-in-cy pipeline. These tests have been flaky for some time and it was difficult to determine what caused the flake since we are unable to see circleCI history that far back without performance implications.Because there are issues recording with
--specspecified with cy-in-cy, we will not be recording the windows app integration tests. Additionally, this fixes thewindows-test-binary-against-kitchensink-chromewhereyarnis not installed on the windows-server executor by making sure yarn is installed first before any action happens when checking out a repo.The launchpad and app tests are still flaky, they are just less flaky and have the ability to pass now. Not ideal, but this is an improvement
Steps to test
How has the user experience changed?
PR Tasks
cypress-documentation?type definitions?