chore: parallelize test-binary-against-recipes CI step #27570
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
astone123
commented
Aug 16, 2023
| @@ -1,5 +1,6 @@ | |||
| { | |||
| "name": "internal-scripts", | |||
| "version": "0.0.0-development", | |||
There was a problem hiding this comment.
This is an un-related drive-by change. There's a warning in the console any time we run any task saying that scripts is missing a version. This is just to make that warning go away
There was a problem hiding this comment.
Been wanting to do this as well
12 flaky tests on run #50046 ↗︎
Details:
|
|||||||||||||||||||||||||||||||||||||||||||||
| Test | Artifacts | |
|---|---|---|
| network stubbing > intercepting request > can delay and throttle a StaticResponse |
Output
Video
|
|
e2e/origin/commands/misc.cy.ts • 1 flaky test • 5x-driver-electron
| Test | Artifacts | |
|---|---|---|
| cy.origin misc > .end() |
Output
Video
|
|
e2e/origin/commands/navigation.cy.ts • 1 flaky test • 5x-driver-electron
| Test | Artifacts | |
|---|---|---|
| cy.origin navigation > .go() |
Output
Video
|
|
e2e/origin/integrity.cy.ts • 1 flaky test • 5x-driver-electron
| Test | Artifacts | |
|---|---|---|
| Integrity Preservation > <script> tags > preserves integrity with static <script> in HTML with SHA256 integrity. |
Output
Video
|
|
e2e/origin/commands/spies_stubs_clocks.cy.ts • 1 flaky test • 5x-driver-electron
| Test | Artifacts | |
|---|---|---|
| cy.origin spies, stubs, and clock > spy() |
Output
Video
|
|
The first 5 flaky specs are shown, see all 10 specs in Cypress Cloud.
This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.
jordanpowell88
approved these changes
Aug 18, 2023
chrisbreiding
approved these changes
Aug 18, 2023
astone123
force-pushed
the
astone123/parallelize-recipes
branch
from
7fb4da0 to
771ded3
Compare
astone123
added a commit
that referenced
this pull request
Aug 18, 2023
astone123
added a commit
that referenced
this pull request
Aug 21, 2023
dkasper-was-taken
added a commit
that referenced
this pull request
Oct 4, 2023
* Revert "chore: simplify build script (#27547)" This reverts commit 0a86ec6. * Revert "chore: upgrade lerna to 6, cache build step (#26913)" This reverts commit 9e60aeb. * [run ci] * chore: upgrade lerna to 6, cache build step (#26913) * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update lerna to 6 * [run ci] * try caching build step * we can't clean without building after * add dependencies on scripts for npm packages * update commands * add config for data-context build step * fix output configurations for npm packages, add gitignores * revert changes to config and data-context build steps * fix outputs * run with cache * fix outputs for cli * actually fix outputs * test with cache --------- Co-authored-by: astone123 <[email protected]> * chore: simplify build script (#27547) * chore: simplify build script * update CI workflows * fix workflows * empty commit because Percy weirdness * chore: add driver, reporter, config as implicit dependencies for runner package (#27559) * run all workflows on branch * chore: parallelize test-binary-against-recipes CI step (#27570) * chore: fix some easy circular dependencies in dep graph (#27612) * chore: remove gulp tasks from postinstall (#27616) * empty commit * chore: minor improvements to node_modules_install (#27647) * chore: fix cypress:open and dev scripts * run with cache [run ci] * exclude mochaawesome assets from .yarnclean [run ci] * bump cache again [run ci] * run cached [run ci] * chore: do not cache cli build step [run ci] * update workflow command and docs for build-cli [run ci] * fix commands that use scope [run ci] * use different branch for publish repo [run ci] * percy weirdness? [run ci] * fix postbuild cli script [run ci] * try to remove typescript from production binary [run ci] * fix circular dependency [run ci] * try removing ts from node_modules [run ci] * remove typescript resolution [run ci] * remove redundant target scripts * update to lerna scoped commands * remove unneeded yarn in lerna command * try to fix Electron install in Windows workflow --------- Co-authored-by: Jordan <[email protected]> Co-authored-by: Dave Kasper <[email protected]>
astone123
added a commit
that referenced
this pull request
Oct 16, 2023
* Revert "chore: simplify build script (#27547)" This reverts commit 0a86ec6. * Revert "chore: upgrade lerna to 6, cache build step (#26913)" This reverts commit 9e60aeb. * [run ci] * chore: upgrade lerna to 6, cache build step (#26913) * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update lerna to 6 * [run ci] * try caching build step * we can't clean without building after * add dependencies on scripts for npm packages * update commands * add config for data-context build step * fix output configurations for npm packages, add gitignores * revert changes to config and data-context build steps * fix outputs * run with cache * fix outputs for cli * actually fix outputs * test with cache --------- Co-authored-by: astone123 <[email protected]> * chore: simplify build script (#27547) * chore: simplify build script * update CI workflows * fix workflows * empty commit because Percy weirdness * chore: add driver, reporter, config as implicit dependencies for runner package (#27559) * run all workflows on branch * chore: parallelize test-binary-against-recipes CI step (#27570) * chore: fix some easy circular dependencies in dep graph (#27612) * chore: remove gulp tasks from postinstall (#27616) * empty commit * chore: minor improvements to node_modules_install (#27647) * chore: fix cypress:open and dev scripts * run with cache [run ci] * exclude mochaawesome assets from .yarnclean [run ci] * bump cache again [run ci] * run cached [run ci] * chore: do not cache cli build step [run ci] * update workflow command and docs for build-cli [run ci] * fix commands that use scope [run ci] * use different branch for publish repo [run ci] * percy weirdness? [run ci] * fix postbuild cli script [run ci] * try to remove typescript from production binary [run ci] * fix circular dependency [run ci] * try removing ts from node_modules [run ci] * remove typescript resolution [run ci] * remove redundant target scripts * update to lerna scoped commands * remove unneeded yarn in lerna command * try to fix Electron install in Windows workflow --------- Co-authored-by: Jordan <[email protected]> Co-authored-by: Dave Kasper <[email protected]>
astone123
added a commit
that referenced
this pull request
Oct 16, 2023
* Revert "chore: simplify build script (#27547)" This reverts commit 0a86ec6. * Revert "chore: upgrade lerna to 6, cache build step (#26913)" This reverts commit 9e60aeb. * [run ci] * chore: upgrade lerna to 6, cache build step (#26913) * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update lerna to 6 * [run ci] * try caching build step * we can't clean without building after * add dependencies on scripts for npm packages * update commands * add config for data-context build step * fix output configurations for npm packages, add gitignores * revert changes to config and data-context build steps * fix outputs * run with cache * fix outputs for cli * actually fix outputs * test with cache --------- Co-authored-by: astone123 <[email protected]> * chore: simplify build script (#27547) * chore: simplify build script * update CI workflows * fix workflows * empty commit because Percy weirdness * chore: add driver, reporter, config as implicit dependencies for runner package (#27559) * run all workflows on branch * chore: parallelize test-binary-against-recipes CI step (#27570) * chore: fix some easy circular dependencies in dep graph (#27612) * chore: remove gulp tasks from postinstall (#27616) * empty commit * chore: minor improvements to node_modules_install (#27647) * chore: fix cypress:open and dev scripts * run with cache [run ci] * exclude mochaawesome assets from .yarnclean [run ci] * bump cache again [run ci] * run cached [run ci] * chore: do not cache cli build step [run ci] * update workflow command and docs for build-cli [run ci] * fix commands that use scope [run ci] * use different branch for publish repo [run ci] * percy weirdness? [run ci] * fix postbuild cli script [run ci] * try to remove typescript from production binary [run ci] * fix circular dependency [run ci] * try removing ts from node_modules [run ci] * remove typescript resolution [run ci] * remove redundant target scripts * update to lerna scoped commands * remove unneeded yarn in lerna command * try to fix Electron install in Windows workflow --------- Co-authored-by: Jordan <[email protected]> Co-authored-by: Dave Kasper <[email protected]>
astone123
added a commit
that referenced
this pull request
Oct 16, 2023
* Revert "chore: simplify build script (#27547)" This reverts commit 0a86ec6. * Revert "chore: upgrade lerna to 6, cache build step (#26913)" This reverts commit 9e60aeb. * [run ci] * chore: upgrade lerna to 6, cache build step (#26913) * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update lerna to 6 * [run ci] * try caching build step * we can't clean without building after * add dependencies on scripts for npm packages * update commands * add config for data-context build step * fix output configurations for npm packages, add gitignores * revert changes to config and data-context build steps * fix outputs * run with cache * fix outputs for cli * actually fix outputs * test with cache --------- Co-authored-by: astone123 <[email protected]> * chore: simplify build script (#27547) * chore: simplify build script * update CI workflows * fix workflows * empty commit because Percy weirdness * chore: add driver, reporter, config as implicit dependencies for runner package (#27559) * run all workflows on branch * chore: parallelize test-binary-against-recipes CI step (#27570) * chore: fix some easy circular dependencies in dep graph (#27612) * chore: remove gulp tasks from postinstall (#27616) * empty commit * chore: minor improvements to node_modules_install (#27647) * chore: fix cypress:open and dev scripts * run with cache [run ci] * exclude mochaawesome assets from .yarnclean [run ci] * bump cache again [run ci] * run cached [run ci] * chore: do not cache cli build step [run ci] * update workflow command and docs for build-cli [run ci] * fix commands that use scope [run ci] * use different branch for publish repo [run ci] * percy weirdness? [run ci] * fix postbuild cli script [run ci] * try to remove typescript from production binary [run ci] * fix circular dependency [run ci] * try removing ts from node_modules [run ci] * remove typescript resolution [run ci] * remove redundant target scripts * update to lerna scoped commands * remove unneeded yarn in lerna command * try to fix Electron install in Windows workflow --------- Co-authored-by: Jordan <[email protected]> Co-authored-by: Dave Kasper <[email protected]>
astone123
added a commit
that referenced
this pull request
Oct 17, 2023
* Revert "chore: simplify build script (#27547)" This reverts commit 0a86ec6. * Revert "chore: upgrade lerna to 6, cache build step (#26913)" This reverts commit 9e60aeb. * [run ci] * chore: upgrade lerna to 6, cache build step (#26913) * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update lerna to 6 * [run ci] * try caching build step * we can't clean without building after * add dependencies on scripts for npm packages * update commands * add config for data-context build step * fix output configurations for npm packages, add gitignores * revert changes to config and data-context build steps * fix outputs * run with cache * fix outputs for cli * actually fix outputs * test with cache --------- Co-authored-by: astone123 <[email protected]> * chore: simplify build script (#27547) * chore: simplify build script * update CI workflows * fix workflows * empty commit because Percy weirdness * chore: add driver, reporter, config as implicit dependencies for runner package (#27559) * run all workflows on branch * chore: parallelize test-binary-against-recipes CI step (#27570) * chore: fix some easy circular dependencies in dep graph (#27612) * chore: remove gulp tasks from postinstall (#27616) * empty commit * chore: minor improvements to node_modules_install (#27647) * chore: fix cypress:open and dev scripts * run with cache [run ci] * exclude mochaawesome assets from .yarnclean [run ci] * bump cache again [run ci] * run cached [run ci] * chore: do not cache cli build step [run ci] * update workflow command and docs for build-cli [run ci] * fix commands that use scope [run ci] * use different branch for publish repo [run ci] * percy weirdness? [run ci] * fix postbuild cli script [run ci] * try to remove typescript from production binary [run ci] * fix circular dependency [run ci] * try removing ts from node_modules [run ci] * remove typescript resolution [run ci] * remove redundant target scripts * update to lerna scoped commands * remove unneeded yarn in lerna command * try to fix Electron install in Windows workflow --------- Co-authored-by: Jordan <[email protected]> Co-authored-by: Dave Kasper <[email protected]>
astone123
added a commit
that referenced
this pull request
Oct 18, 2023
* Revert "chore: simplify build script (#27547)" This reverts commit 0a86ec6. * Revert "chore: upgrade lerna to 6, cache build step (#26913)" This reverts commit 9e60aeb. * [run ci] * chore: upgrade lerna to 6, cache build step (#26913) * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update lerna to 6 * [run ci] * try caching build step * we can't clean without building after * add dependencies on scripts for npm packages * update commands * add config for data-context build step * fix output configurations for npm packages, add gitignores * revert changes to config and data-context build steps * fix outputs * run with cache * fix outputs for cli * actually fix outputs * test with cache --------- Co-authored-by: astone123 <[email protected]> * chore: simplify build script (#27547) * chore: simplify build script * update CI workflows * fix workflows * empty commit because Percy weirdness * chore: add driver, reporter, config as implicit dependencies for runner package (#27559) * run all workflows on branch * chore: parallelize test-binary-against-recipes CI step (#27570) * chore: fix some easy circular dependencies in dep graph (#27612) * chore: remove gulp tasks from postinstall (#27616) * empty commit * chore: minor improvements to node_modules_install (#27647) * chore: fix cypress:open and dev scripts * run with cache [run ci] * exclude mochaawesome assets from .yarnclean [run ci] * bump cache again [run ci] * run cached [run ci] * chore: do not cache cli build step [run ci] * update workflow command and docs for build-cli [run ci] * fix commands that use scope [run ci] * use different branch for publish repo [run ci] * percy weirdness? [run ci] * fix postbuild cli script [run ci] * try to remove typescript from production binary [run ci] * fix circular dependency [run ci] * try removing ts from node_modules [run ci] * remove typescript resolution [run ci] * remove redundant target scripts * update to lerna scoped commands * remove unneeded yarn in lerna command * try to fix Electron install in Windows workflow --------- Co-authored-by: Jordan <[email protected]> Co-authored-by: Dave Kasper <[email protected]>
astone123
added a commit
that referenced
this pull request
Oct 18, 2023
* Revert "chore: simplify build script (#27547)" This reverts commit 0a86ec6. * Revert "chore: upgrade lerna to 6, cache build step (#26913)" This reverts commit 9e60aeb. * [run ci] * chore: upgrade lerna to 6, cache build step (#26913) * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update lerna to 6 * [run ci] * try caching build step * we can't clean without building after * add dependencies on scripts for npm packages * update commands * add config for data-context build step * fix output configurations for npm packages, add gitignores * revert changes to config and data-context build steps * fix outputs * run with cache * fix outputs for cli * actually fix outputs * test with cache --------- Co-authored-by: astone123 <[email protected]> * chore: simplify build script (#27547) * chore: simplify build script * update CI workflows * fix workflows * empty commit because Percy weirdness * chore: add driver, reporter, config as implicit dependencies for runner package (#27559) * run all workflows on branch * chore: parallelize test-binary-against-recipes CI step (#27570) * chore: fix some easy circular dependencies in dep graph (#27612) * chore: remove gulp tasks from postinstall (#27616) * empty commit * chore: minor improvements to node_modules_install (#27647) * chore: fix cypress:open and dev scripts * run with cache [run ci] * exclude mochaawesome assets from .yarnclean [run ci] * bump cache again [run ci] * run cached [run ci] * chore: do not cache cli build step [run ci] * update workflow command and docs for build-cli [run ci] * fix commands that use scope [run ci] * use different branch for publish repo [run ci] * percy weirdness? [run ci] * fix postbuild cli script [run ci] * try to remove typescript from production binary [run ci] * fix circular dependency [run ci] * try removing ts from node_modules [run ci] * remove typescript resolution [run ci] * remove redundant target scripts * update to lerna scoped commands * remove unneeded yarn in lerna command * try to fix Electron install in Windows workflow --------- Co-authored-by: Jordan <[email protected]> Co-authored-by: Dave Kasper <[email protected]>
astone123
added a commit
that referenced
this pull request
Oct 18, 2023
* Revert "chore: simplify build script (#27547)" This reverts commit 0a86ec6. * Revert "chore: upgrade lerna to 6, cache build step (#26913)" This reverts commit 9e60aeb. * [run ci] * chore: upgrade lerna to 6, cache build step (#26913) * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update lerna to 6 * [run ci] * try caching build step * we can't clean without building after * add dependencies on scripts for npm packages * update commands * add config for data-context build step * fix output configurations for npm packages, add gitignores * revert changes to config and data-context build steps * fix outputs * run with cache * fix outputs for cli * actually fix outputs * test with cache --------- Co-authored-by: astone123 <[email protected]> * chore: simplify build script (#27547) * chore: simplify build script * update CI workflows * fix workflows * empty commit because Percy weirdness * chore: add driver, reporter, config as implicit dependencies for runner package (#27559) * run all workflows on branch * chore: parallelize test-binary-against-recipes CI step (#27570) * chore: fix some easy circular dependencies in dep graph (#27612) * chore: remove gulp tasks from postinstall (#27616) * empty commit * chore: minor improvements to node_modules_install (#27647) * chore: fix cypress:open and dev scripts * run with cache [run ci] * exclude mochaawesome assets from .yarnclean [run ci] * bump cache again [run ci] * run cached [run ci] * chore: do not cache cli build step [run ci] * update workflow command and docs for build-cli [run ci] * fix commands that use scope [run ci] * use different branch for publish repo [run ci] * percy weirdness? [run ci] * fix postbuild cli script [run ci] * try to remove typescript from production binary [run ci] * fix circular dependency [run ci] * try removing ts from node_modules [run ci] * remove typescript resolution [run ci] * remove redundant target scripts * update to lerna scoped commands * remove unneeded yarn in lerna command * try to fix Electron install in Windows workflow --------- Co-authored-by: Jordan <[email protected]> Co-authored-by: Dave Kasper <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Additional details
According to CircleCI insights,
test-binary-against-recipesis the longest-running job that we have in our CI pipeline (it only runs ondevelopand other branches specified inmainBuildFilters). It's p95 runtime for the past 30 days is 25m 14s. Because it runs in parallel with the rest of the binary tasks, we need to wait for it to complete to release Cypress, re-run the workflow from failed, etc. This adds a lot of time to our development cycle.This PR uses the existing chunking logic in the
test-examples.jsscript in cypress-example-recipes to split these tests across 4 machines. This brings the runtime in line with the rest of the binary steps, so we're not over-optimizing, just optimizing enough so that the runtime for all of the binary steps are within 1 minute or so of eachother.This should significantly decrease the average runtime for our
linux-x64workflow ondevelop.Steps to test
Look at the CI runs for this branch and specifically the
test-binary-against-recipesjob.How has the user experience changed?
n/a
PR Tasks
cypress-documentation?type definitions?