chore: optimize task execution #27848

astone123 · 2023-09-19T23:07:59Z

This PR makes various improvements to task execution both locally and in CI. Main changes are

Allow Lerna (Nx) to cache the results of build targets for all packages (note that the CLI package cypress is excluded from this and uses its own command build-cli)
Remove buildProd gulp command and incorporate steps into build commands inside of the packages that are related to each step. This allows for caching these steps with Lerna
Change the name of the cli build step to build-cli so that Lerna doesn't cache it
Add yarn autoclean to remove unnecessary files from node modules to reduce the cache size
Remove unnecessary build-prod targets for packages
Parallelize test runs for test-binary-against-recipes job in CI
Resolve various circular dependencies between packages

The results of these changes are:

Decreased CI runtime for most change sets
Decreased build time for local development

Testing

Download a pre-release binary from a recent commit and manually smoke test it.
Look at the CI runs for this branch and look for abnormal behavior

Related work

This work requires changes to the cypress-publish-binary project as well
https://github.com/cypress-io/cypress-publish-binary/pull/28

This reverts commit 0a86ec6.

This reverts commit 9e60aeb.

* chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update build-npm-modules script * chore: update lerna to 6 * [run ci] * try caching build step * we can't clean without building after * add dependencies on scripts for npm packages * update commands * add config for data-context build step * fix output configurations for npm packages, add gitignores * revert changes to config and data-context build steps * fix outputs * run with cache * fix outputs for cli * actually fix outputs * test with cache --------- Co-authored-by: astone123 <[email protected]>

* chore: simplify build script * update CI workflows * fix workflows * empty commit because Percy weirdness

…er package (#27559)

astone123 · 2023-09-20T16:01:57Z

scripts/binary/trigger-publish-binary-pipeline.js

@@ -14,6 +14,7 @@ const { getNextVersionForBinary } = require('../get-next-version')
  }

  const body = JSON.stringify({
+    branch: 'lerna-optimize-tasks',


This ensures that the pipeline that we trigger runs on the lerna-optimize-tasks branch in the publish binary workflow. I'll remove this before merging and merge my corresponding PR https://github.com/cypress-io/cypress-publish-binary/pull/28

astone123 · 2023-09-20T16:02:57Z

scripts/binary/build.ts

@@ -102,7 +102,12 @@ export async function buildCypressApp (options: BuildCypressAppOpts) {
  if (!keepBuild) {
    log('#buildPackages')

-    await execa('yarn', ['lerna', 'run', 'build-prod', '--ignore', 'cli', '--concurrency', '4'], {
+    await execa('yarn', ['lerna', 'run', 'build', '--concurrency', '4'], {


We want to make sure that everything is built here before we build the binary. These will be cached because we already built them in a previous step. We can probably remove this in the future

astone123 · 2023-09-20T16:04:20Z

scripts/gulp/gulpfile.ts

@@ -116,30 +115,6 @@ gulp.task('open', startCypressWatch)
 *  Tasks that aren't watched. Usually composed together with other tasks.
 *------------------------------------------------------------------------**/

-gulp.task('buildProd',


I targeted this task because it was being called in CI, but I want to come back and clean up more of these tasks - we don't need to use Gulp for these tasks. Each task should be isolated to its related package if possible

astone123 · 2023-09-20T16:04:49Z

scripts/package.json

@@ -1,5 +1,6 @@
 {
  "name": "internal-scripts",
+  "version": "0.0.0-development",


This gets rid of that annoying warning saying that scripts doesn't have a version

cypress · 2023-09-20T17:01:50Z

5 flaky tests on run #51443 ↗︎

0	466	19	0	5

Details:

try to fix Electron install in Windows workflow
Project: cypress	Commit: `0caf3fd52a`
Status: Passed	Duration: 16:50 💡
Started: Oct 4, 2023 5:11 PM	Ended: Oct 4, 2023 5:28 PM

cypress\e2e\runs.cy.ts • 1 flaky test • app-e2e

View Output Video

Test		Artifacts
... > displays a list of recorded runs if a run has been recorded		`Test Replay` `Output` `Screenshots`

cypress\e2e\create-from-component.cy.ts • 1 flaky test • app-e2e

View Output Video

Test		Artifacts
... > Shows create from component card for React projects		`Test Replay` `Output` `Screenshots`

cypress\e2e\specs.cy.ts • 1 flaky test • app-e2e

View Output Video

Test		Artifacts
... > shows No Specs page with specPattern from config		`Test Replay` `Output` `Screenshots`

cypress\e2e\specs_list_latest_runs.cy.ts • 2 flaky tests • app-e2e

View Output Video

Test		Artifacts
App/Cloud Integration - Latest runs and Average duration > when no runs are recorded > shows placeholders for all visible specs		`Test Replay` `Output` `Screenshots`
App/Cloud Integration - Latest runs and Average duration > when offline > shows offline alert then hides it after coming online		`Test Replay` `Output` `Screenshots`

Review all test suite changes for PR #27848 ↗︎

jordanpowell88

Looks good to me overall. Left a couple of minor changes that aren't show stoppers but ideally we can resolve them before merging. Also left a few future improvements comments along the way that we can use to inform future improvements

npm/react/package.json

npm/react18/package.json

jordanpowell88 · 2023-10-02T22:20:35Z

packages/graphql/package.json

+          "nexus-build"
+        ],
+        "outputs": [
+          "{workspaceRoot}/packages/frontend-shared/cypress/support/generated/test-graphql-types.gen.ts",


It's worth noting that we should eventually move away from one lib having outputs in another. It's fine for now but we should consider refactoring this at some point

jordanpowell88 · 2023-10-02T22:28:15Z

scripts/gulp/gulpfile.ts

@@ -170,9 +145,6 @@ gulp.task('watchForE2E', gulp.series(
 *------------------------------------------------------------------------**/

 gulp.task('cyRunLaunchpadE2E', gulp.series(


Is this no longer needed because the build has already been cached? This probably works as is in CI but ideally those gulp script that call the build-prod would have a depends on build in case someone tries to run this locally. Probably ok for now but we want to get to a place where local and CI run the same way

scripts/run-postInstall.js

cypress-bot · 2023-10-11T23:30:02Z

Released in 13.3.1.

This comment thread has been locked. If you are still experiencing this issue after upgrading to
Cypress v13.3.1, please open a new issue.

astone123 and others added 29 commits August 21, 2023 11:36

Revert "chore: simplify build script (#27547)"

29cd85c

This reverts commit 0a86ec6.

Revert "chore: upgrade lerna to 6, cache build step (#26913)"

54285be

This reverts commit 9e60aeb.

[run ci]

bff487d

chore: simplify build script (#27547)

e8fee7e

* chore: simplify build script * update CI workflows * fix workflows * empty commit because Percy weirdness

chore: add driver, reporter, config as implicit dependencies for runn…

ec1c09f

…er package (#27559)

run all workflows on branch

f76b3de

chore: parallelize test-binary-against-recipes CI step (#27570)

143b23f

chore: fix some easy circular dependencies in dep graph (#27612)

23559cc

chore: remove gulp tasks from postinstall (#27616)

1de6d84

empty commit

0263696

chore: minor improvements to node_modules_install (#27647)

53c9c90

chore: fix cypress:open and dev scripts

6caa15b

run with cache [run ci]

efec954

exclude mochaawesome assets from .yarnclean [run ci]

7af9f61

bump cache again [run ci]

c6aefd6

Merge branch 'develop' into lerna-optimize-tasks

54046a9

run cached [run ci]

9bc5344

chore: do not cache cli build step [run ci]

414d3ba

update workflow command and docs for build-cli [run ci]

c2ee0dc

fix commands that use scope [run ci]

94d36f1

use different branch for publish repo [run ci]

724f407

percy weirdness? [run ci]

8cb63cf

fix postbuild cli script [run ci]

8f47466

Merge branch 'develop' into lerna-optimize-tasks

9be1b89

try to remove typescript from production binary [run ci]

f3ea19f

fix circular dependency [run ci]

918dcc2

try removing ts from node_modules [run ci]

40d92d4

remove typescript resolution [run ci]

29fea42

astone123 self-assigned this Sep 19, 2023

Merge branch 'develop' into lerna-optimize-tasks

8b5e904

astone123 changed the title ~~optimize tasks~~ chore: optimize task execution Sep 20, 2023

astone123 commented Sep 20, 2023

View reviewed changes

astone123 marked this pull request as ready for review September 20, 2023 16:05

astone123 requested review from mjhenkes, mschile, jordanpowell88, AtofStryker and ryanthemanuel September 20, 2023 18:53

nagash77 requested a review from dkasper-was-taken September 26, 2023 18:22

Merge branch 'develop' into lerna-optimize-tasks

28e904d

dkasper-was-taken approved these changes Sep 26, 2023

View reviewed changes

Merge branch 'develop' into lerna-optimize-tasks

6dd0045

jordanpowell88 approved these changes Oct 2, 2023

View reviewed changes

dkasper-was-taken and others added 5 commits October 3, 2023 16:10

Merge branch 'develop' into lerna-optimize-tasks

dfec4ad

remove redundant target scripts

ea62272

update to lerna scoped commands

39981ce

remove unneeded yarn in lerna command

1e066de

try to fix Electron install in Windows workflow

0caf3fd

dkasper-was-taken merged commit 4c11731 into develop Oct 4, 2023
9 checks passed

dkasper-was-taken deleted the lerna-optimize-tasks branch October 4, 2023 17:25

dkasper-was-taken mentioned this pull request Oct 4, 2023

chore: follow-up cli optimizations #27974

Merged

cypress-bot bot locked as resolved and limited conversation to collaborators Oct 11, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: optimize task execution #27848

chore: optimize task execution #27848

astone123 commented Sep 19, 2023 •

edited

Loading

astone123 Sep 20, 2023

astone123 Sep 20, 2023

astone123 Sep 20, 2023

astone123 Sep 20, 2023

cypress bot commented Sep 20, 2023 •

edited

Loading

jordanpowell88 left a comment

jordanpowell88 Oct 2, 2023

jordanpowell88 Oct 2, 2023

cypress-bot bot commented Oct 11, 2023

		@@ -170,9 +145,6 @@ gulp.task('watchForE2E', gulp.series(
		------------------------------------------------------------------------*/

		gulp.task('cyRunLaunchpadE2E', gulp.series(

chore: optimize task execution #27848

chore: optimize task execution #27848

Conversation

astone123 commented Sep 19, 2023 • edited Loading

Testing

Related work

astone123 Sep 20, 2023

Choose a reason for hiding this comment

astone123 Sep 20, 2023

Choose a reason for hiding this comment

astone123 Sep 20, 2023

Choose a reason for hiding this comment

astone123 Sep 20, 2023

Choose a reason for hiding this comment

cypress bot commented Sep 20, 2023 • edited Loading

5 flaky tests on run #51443 ↗︎

Review all test suite changes for PR #27848 ↗︎

jordanpowell88 left a comment

Choose a reason for hiding this comment

jordanpowell88 Oct 2, 2023

Choose a reason for hiding this comment

jordanpowell88 Oct 2, 2023

Choose a reason for hiding this comment

cypress-bot bot commented Oct 11, 2023

astone123 commented Sep 19, 2023 •

edited

Loading

cypress bot commented Sep 20, 2023 •

edited

Loading