[security_solution] Cypress flaky tests catcher #162376
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…opycinski/kibana into chore/cypress-flaky-tests-runner
…laky-tests-runner # Conflicts: # .buildkite/pipelines/pull_request/security_solution.yml # package.json # x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts # yarn.lock
patrykkopycinski
changed the title
jloleysens
approved these changes
Aug 8, 2023
|
@elasticmachine test this please |
patrykkopycinski
added
release_note:skip
|
buildkite test this |
patrykkopycinski
added
release_note:skip
|
buildkite test this |
…laky-tests-runner # Conflicts: # x-pack/plugins/security_solution/package.json
💚 Build Succeeded
Metrics [docs]Unknown metric groupsESLint disabled line counts
Total ESLint disabled count
History
To update your PR or re-run it, just comment with: |
kibanamachine
added
v8.10.0
backport:skip
|
@patrykkopycinski I think we can now backport this into the 7.17 branch after the work we have done at #163363 |
patrykkopycinski
added a commit
to patrykkopycinski/kibana
that referenced
this pull request
Aug 15, 2023
## Summary Inspired by https://glebbahmutov.com/blog/burning-tests/ Implements the idea presented here https://glebbahmutov.com/blog/burning-tests/#bonus-3-burning-new-or-changed-specs In short, on PR that is changing/adding new Cypress spec files we will try to "burn" them, it means we will try to run each `it` `2` times to make sure tests are written in a way that gives Cypress a chance to recover from the failed test. Also adding a command that allows to "burn" tests locally ``` yarn cypress:burn --spec "<>" ``` Right now the job is set to `soft_fail`, so it is not going to block the PR from merging, but hopefully will help the Team to recognize potential flakiness before it is merged to `main` (cherry picked from commit 09aaecb) # Conflicts: # package.json # typings/index.d.ts # x-pack/plugins/security_solution/cypress/scripts/run_cypress/parallel.ts # x-pack/plugins/security_solution/cypress/support/e2e.js # x-pack/plugins/security_solution/package.json # yarn.lock
patrykkopycinski
added a commit
that referenced
this pull request
Aug 22, 2023
…3992) # Backport This will backport the following commits from `main` to `7.17`: - [[security_solution] Cypress flaky tests catcher (#162376)](#162376) <!--- Backport version: 8.9.8 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Patryk Kopyciński","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-08-09T21:58:04Z","message":"[security_solution] Cypress flaky tests catcher (#162376)\n\n## Summary\r\n\r\nInspired by https://glebbahmutov.com/blog/burning-tests/\r\n\r\nImplements the idea presented here\r\nhttps://glebbahmutov.com/blog/burning-tests/#bonus-3-burning-new-or-changed-specs\r\n\r\nIn short, on PR that is changing/adding new Cypress spec files we will\r\ntry to \"burn\" them, it means we will try to run each `it` `2` times to\r\nmake sure tests are written in a way that gives Cypress a chance to\r\nrecover from the failed test.\r\nAlso adding a command that allows to \"burn\" tests locally\r\n```\r\nyarn cypress:burn --spec \"<>\"\r\n```\r\n\r\nRight now the job is set to `soft_fail`, so it is not going to block the\r\nPR from merging, but hopefully will help the Team to recognize potential\r\nflakiness before it is merged to `main`","sha":"09aaecb59d5e82e17bf6f274de3cedd394bbe25b","branchLabelMapping":{"^v8.10.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:skip","v8.10.0"],"number":162376,"url":"https://github.com/elastic/kibana/pull/162376","mergeCommit":{"message":"[security_solution] Cypress flaky tests catcher (#162376)\n\n## Summary\r\n\r\nInspired by https://glebbahmutov.com/blog/burning-tests/\r\n\r\nImplements the idea presented here\r\nhttps://glebbahmutov.com/blog/burning-tests/#bonus-3-burning-new-or-changed-specs\r\n\r\nIn short, on PR that is changing/adding new Cypress spec files we will\r\ntry to \"burn\" them, it means we will try to run each `it` `2` times to\r\nmake sure tests are written in a way that gives Cypress a chance to\r\nrecover from the failed test.\r\nAlso adding a command that allows to \"burn\" tests locally\r\n```\r\nyarn cypress:burn --spec \"<>\"\r\n```\r\n\r\nRight now the job is set to `soft_fail`, so it is not going to block the\r\nPR from merging, but hopefully will help the Team to recognize potential\r\nflakiness before it is merged to `main`","sha":"09aaecb59d5e82e17bf6f274de3cedd394bbe25b"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.10.0","labelRegex":"^v8.10.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/162376","number":162376,"mergeCommit":{"message":"[security_solution] Cypress flaky tests catcher (#162376)\n\n## Summary\r\n\r\nInspired by https://glebbahmutov.com/blog/burning-tests/\r\n\r\nImplements the idea presented here\r\nhttps://glebbahmutov.com/blog/burning-tests/#bonus-3-burning-new-or-changed-specs\r\n\r\nIn short, on PR that is changing/adding new Cypress spec files we will\r\ntry to \"burn\" them, it means we will try to run each `it` `2` times to\r\nmake sure tests are written in a way that gives Cypress a chance to\r\nrecover from the failed test.\r\nAlso adding a command that allows to \"burn\" tests locally\r\n```\r\nyarn cypress:burn --spec \"<>\"\r\n```\r\n\r\nRight now the job is set to `soft_fail`, so it is not going to block the\r\nPR from merging, but hopefully will help the Team to recognize potential\r\nflakiness before it is merged to `main`","sha":"09aaecb59d5e82e17bf6f274de3cedd394bbe25b"}}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <[email protected]> Co-authored-by: Tiago Costa <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Inspired by https://glebbahmutov.com/blog/burning-tests/
Implements the idea presented here https://glebbahmutov.com/blog/burning-tests/#bonus-3-burning-new-or-changed-specs
In short, on PR that is changing/adding new Cypress spec files we will try to "burn" them, it means we will try to run each
it2times to make sure tests are written in a way that gives Cypress a chance to recover from the failed test.Also adding a command that allows to "burn" tests locally
Right now the job is set to
soft_fail, so it is not going to block the PR from merging, but hopefully will help the Team to recognize potential flakiness before it is merged tomain