[Security Solution] Enable Cypress tests in Serverless: Rule Management area #161540
Assignees
Labels
8.12 candidate
Team:Detection Rule Management
Security Detection Rule Management Team
Team:Detections and Resp
Security Detection Response Team
Team: SecuritySolution
Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
test
test-coverage
issues & PRs for improving code test coverage
Comments
banderror
changed the title
banderror
added
test
test-coverage
banderror
changed the title
This was referenced Sep 1, 2023
banderror
added a commit
that referenced
this issue
Sep 9, 2023
…rverless (#165966) **Resolves: #164441 ## Summary Skips all Cypress tests owned by @elastic/security-detection-rule-management and @elastic/security-detection-engine teams in Serverless using the new `@skipInServerless` tag. - Adds a new `@skipInServerless` tag - Updates `x-pack/test/security_solution_cypress/cypress/README.md` to explain when to use what tags - Explicitly adds missing tags to all D&R tests - Adds `// TODO:` comments to tests with links to follow-up tickets - Fixes the `x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts` script (see below) Follow-up work: - #161540 - #161539 ## Context > Serverless test failures will soon block PR merge > During the development of the serverless test infrastructure, we decided that serverless tests will only soft-fail. That means you see the test failure in your PR but you're still able to merge. We did that mainly in order to not block delivery of stateful features and bug fixes while serverless tests and pipelines were implemented and stabilized. We now have the major building blocks for the serverless test infrastructure in place and will integrate serverless tests in our regular pipelines. As part of this process, we're skipping failing and flaky serverless tests that came in during the soft-fail phase. Once this is done, a PR with serverless test failures can no longer be merged, similar to how we have it for stateful test failures. > We plan to merge this in the next few days and we'll send out another notification when it's done. ## Fixing `parallel.ts` There were two problems with the `x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts` script we use for running Cypress tests: - The script was broken in #162673 (here on [this line](https://github.com/elastic/kibana/pull/162673/files#diff-9f40ced6d29c4fc2709af881680400293d8ce1bc9ebb07b9138d6d99c83c09c9R67)) - I think it has never supported situations when all tests matching a spec pattern (such as `./cypress/e2e/!(investigations|explore)/**/*.cy.ts`) end up being skipped via Cypress tags (such as `@skipInServerless`) Both the issues are fixed in this PR. Code owners are added for this script in the CODEOWNERS file to prevent breaking this script in future PRs.
This was referenced Sep 21, 2023
maximpn
added a commit
that referenced
this issue
Sep 21, 2023
**Related to:** #161540, #161539 ## Summary Always force the last breadcrumb to be inactive. ## Details Usual UX expects the last breadcrumb to be inactive as it represents the current page. The same can be seen from EUI [examples](https://eui.elastic.co/#/navigation/breadcrumbs). It turns out Serverless Security Solution plugin does't remove `href` and `onClick` fields from the last breadcrumb and passes it to `chrome.setBreadcrumbs()` or `serverless.setBreadcrumbs()` which renders the last breadcrumb as active but clicking on it only refreshes the page. ESS Security Solution on the other hand processes breadcrumbs currently. The same behavior may be the case for the other plungs as well. As it's much simpler to strip off undesired fields at one place instead of processing them in plugins it's done in `packages/core/chrome/core-chrome-browser-internal/src/ui/header/header_breadcrumbs.tsx`. Security Solution codebase has been updated accordingly. A side effect of this PR is consistent ESS and Serverless breadcrumbs behavior and it will help to reuse ESS tests for Serverless.
banderror
pushed a commit
that referenced
this issue
Sep 28, 2023
…166771) **Relates to:** #161540 ## Summary This PR unskips rules table filtering serverless tests. Serverless [rules_table_filtering.cy.ts (100 runs)](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3205) 🟢
maximpn
added a commit
that referenced
this issue
Oct 6, 2023
**Addresses:** #161540 ## Summary This PR unskips preuilt rules Cypress tests (besides `prebuilt_rules_install_update_authorization.cy.ts` as it uses non default role). ## Flaky test runner [prebuilt_rules folder 150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3384) 🟢
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Oct 6, 2023
**Addresses:** elastic#161540 ## Summary This PR unskips preuilt rules Cypress tests (besides `prebuilt_rules_install_update_authorization.cy.ts` as it uses non default role). ## Flaky test runner [prebuilt_rules folder 150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3384) 🟢 (cherry picked from commit 6c5603c)
kibanamachine
referenced
this issue
Oct 6, 2023
…) (#168292) # Backport This will backport the following commits from `main` to `8.11`: - [[Security Solution] Unskip prebuilt rules Cypress tests (#168165)](#168165) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-10-06T20:05:43Z","message":"[Security Solution] Unskip prebuilt rules Cypress tests (#168165)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips preuilt rules Cypress tests (besides `prebuilt_rules_install_update_authorization.cy.ts` as it uses non default role).\r\n\r\n## Flaky test runner\r\n\r\n[prebuilt_rules folder 150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3384) 🟢","sha":"6c5603c95223f5ae9883195c7a5ed126513932c1","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","v8.11.0","v8.12.0"],"number":168165,"url":"https://github.com/elastic/kibana/pull/168165","mergeCommit":{"message":"[Security Solution] Unskip prebuilt rules Cypress tests (#168165)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips preuilt rules Cypress tests (besides `prebuilt_rules_install_update_authorization.cy.ts` as it uses non default role).\r\n\r\n## Flaky test runner\r\n\r\n[prebuilt_rules folder 150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3384) 🟢","sha":"6c5603c95223f5ae9883195c7a5ed126513932c1"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/168165","number":168165,"mergeCommit":{"message":"[Security Solution] Unskip prebuilt rules Cypress tests (#168165)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips preuilt rules Cypress tests (besides `prebuilt_rules_install_update_authorization.cy.ts` as it uses non default role).\r\n\r\n## Flaky test runner\r\n\r\n[prebuilt_rules folder 150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3384) 🟢","sha":"6c5603c95223f5ae9883195c7a5ed126513932c1"}}]}] BACKPORT--> Co-authored-by: Maxim Palenov <[email protected]>
banderror
pushed a commit
that referenced
this issue
Oct 9, 2023
) **Addresses:** #161540 ## Summary This PR unskips rules table Serverless Cypress tests - `rules_table_auto_refresh.cy.ts` - `rules_table_links.cy.ts` - `rules_table_persistent_state.cy.ts` - `rules_table_selection.cy.ts` - `rules_table_sorting.cy.ts` ## Flaky test runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3406 150 runs 🟢
banderror
pushed a commit
that referenced
this issue
Oct 9, 2023
…sts (#168305) **Addresses:** #161540 ## Summary This PR unskips miscelanious (not falling in one category but unskipping each one separately is overkill) rule management Serverless Cypress tests - `maintenance_window_callout.cy.ts` - `related_integrations.cy.ts` - `rule_delete.cy.ts` - `rule_snoozing.cy.ts` ## Flaky test runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3405 150 runs 🟢
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Oct 9, 2023
…tic#168306) **Addresses:** elastic#161540 ## Summary This PR unskips rules table Serverless Cypress tests - `rules_table_auto_refresh.cy.ts` - `rules_table_links.cy.ts` - `rules_table_persistent_state.cy.ts` - `rules_table_selection.cy.ts` - `rules_table_sorting.cy.ts` ## Flaky test runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3406 150 runs 🟢 (cherry picked from commit feb4222)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Oct 9, 2023
…sts (elastic#168305) **Addresses:** elastic#161540 ## Summary This PR unskips miscelanious (not falling in one category but unskipping each one separately is overkill) rule management Serverless Cypress tests - `maintenance_window_callout.cy.ts` - `related_integrations.cy.ts` - `rule_delete.cy.ts` - `rule_snoozing.cy.ts` ## Flaky test runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3405 150 runs 🟢 (cherry picked from commit b7116b1)
kibanamachine
referenced
this issue
Oct 9, 2023
#168306) (#168407) # Backport This will backport the following commits from `main` to `8.11`: - [[Security Solution] Unskip rules table Serverless Cypress tests (#168306)](#168306) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-10-09T20:29:22Z","message":"[Security Solution] Unskip rules table Serverless Cypress tests (#168306)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips rules table Serverless Cypress tests\r\n\r\n- `rules_table_auto_refresh.cy.ts`\r\n- `rules_table_links.cy.ts`\r\n- `rules_table_persistent_state.cy.ts`\r\n- `rules_table_selection.cy.ts`\r\n- `rules_table_sorting.cy.ts`\r\n\r\n## Flaky test runner\r\n\r\n\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3406\r\n150 runs 🟢","sha":"feb4222bf6207f71122fd40a96872b3b3e8a84dc","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","v8.11.0","v8.12.0"],"number":168306,"url":"https://github.com/elastic/kibana/pull/168306","mergeCommit":{"message":"[Security Solution] Unskip rules table Serverless Cypress tests (#168306)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips rules table Serverless Cypress tests\r\n\r\n- `rules_table_auto_refresh.cy.ts`\r\n- `rules_table_links.cy.ts`\r\n- `rules_table_persistent_state.cy.ts`\r\n- `rules_table_selection.cy.ts`\r\n- `rules_table_sorting.cy.ts`\r\n\r\n## Flaky test runner\r\n\r\n\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3406\r\n150 runs 🟢","sha":"feb4222bf6207f71122fd40a96872b3b3e8a84dc"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/168306","number":168306,"mergeCommit":{"message":"[Security Solution] Unskip rules table Serverless Cypress tests (#168306)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips rules table Serverless Cypress tests\r\n\r\n- `rules_table_auto_refresh.cy.ts`\r\n- `rules_table_links.cy.ts`\r\n- `rules_table_persistent_state.cy.ts`\r\n- `rules_table_selection.cy.ts`\r\n- `rules_table_sorting.cy.ts`\r\n\r\n## Flaky test runner\r\n\r\n\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3406\r\n150 runs 🟢","sha":"feb4222bf6207f71122fd40a96872b3b3e8a84dc"}}]}] BACKPORT--> Co-authored-by: Maxim Palenov <[email protected]>
kibanamachine
referenced
this issue
Oct 9, 2023
…ress tests (#168305) (#168408) # Backport This will backport the following commits from `main` to `8.11`: - [[Security Solution] Unskip misc rule management Serverless Cypress tests (#168305)](#168305) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-10-09T20:33:22Z","message":"[Security Solution] Unskip misc rule management Serverless Cypress tests (#168305)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips miscelanious (not falling in one category but unskipping\r\neach one separately is overkill) rule management Serverless Cypress\r\ntests\r\n\r\n- `maintenance_window_callout.cy.ts`\r\n- `related_integrations.cy.ts`\r\n- `rule_delete.cy.ts`\r\n- `rule_snoozing.cy.ts`\r\n \r\n## Flaky test runner\r\n\r\n\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3405\r\n150 runs 🟢","sha":"b7116b1efb96f0cedd8afb6529b8186a9ccdaff2","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","v8.11.0","v8.12.0"],"number":168305,"url":"https://github.com/elastic/kibana/pull/168305","mergeCommit":{"message":"[Security Solution] Unskip misc rule management Serverless Cypress tests (#168305)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips miscelanious (not falling in one category but unskipping\r\neach one separately is overkill) rule management Serverless Cypress\r\ntests\r\n\r\n- `maintenance_window_callout.cy.ts`\r\n- `related_integrations.cy.ts`\r\n- `rule_delete.cy.ts`\r\n- `rule_snoozing.cy.ts`\r\n \r\n## Flaky test runner\r\n\r\n\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3405\r\n150 runs 🟢","sha":"b7116b1efb96f0cedd8afb6529b8186a9ccdaff2"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/168305","number":168305,"mergeCommit":{"message":"[Security Solution] Unskip misc rule management Serverless Cypress tests (#168305)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips miscelanious (not falling in one category but unskipping\r\neach one separately is overkill) rule management Serverless Cypress\r\ntests\r\n\r\n- `maintenance_window_callout.cy.ts`\r\n- `related_integrations.cy.ts`\r\n- `rule_delete.cy.ts`\r\n- `rule_snoozing.cy.ts`\r\n \r\n## Flaky test runner\r\n\r\n\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3405\r\n150 runs 🟢","sha":"b7116b1efb96f0cedd8afb6529b8186a9ccdaff2"}}]}] BACKPORT--> Co-authored-by: Maxim Palenov <[email protected]>
dej611
pushed a commit
to dej611/kibana
that referenced
this issue
Oct 17, 2023
**Addresses:** elastic#161540 ## Summary This PR unskips preuilt rules Cypress tests (besides `prebuilt_rules_install_update_authorization.cy.ts` as it uses non default role). ## Flaky test runner [prebuilt_rules folder 150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3384) 🟢
dej611
pushed a commit
to dej611/kibana
that referenced
this issue
Oct 17, 2023
…tic#168306) **Addresses:** elastic#161540 ## Summary This PR unskips rules table Serverless Cypress tests - `rules_table_auto_refresh.cy.ts` - `rules_table_links.cy.ts` - `rules_table_persistent_state.cy.ts` - `rules_table_selection.cy.ts` - `rules_table_sorting.cy.ts` ## Flaky test runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3406 150 runs 🟢
dej611
pushed a commit
to dej611/kibana
that referenced
this issue
Oct 17, 2023
…sts (elastic#168305) **Addresses:** elastic#161540 ## Summary This PR unskips miscelanious (not falling in one category but unskipping each one separately is overkill) rule management Serverless Cypress tests - `maintenance_window_callout.cy.ts` - `related_integrations.cy.ts` - `rule_delete.cy.ts` - `rule_snoozing.cy.ts` ## Flaky test runner https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3405 150 runs 🟢
This was referenced Oct 18, 2023
maximpn
added a commit
that referenced
this issue
Oct 18, 2023
…ts (#169230) **Addresses:** #161540 ## Summary This PR unskips import and export security rules Serverless Cypress tests. ## Flaky test runner `import_export` folder [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3588)
maximpn
added a commit
that referenced
this issue
Oct 18, 2023
#168307) **Addresses:** #161540 ## Summary This PR unskips rules bulk actions Serverless Cypress tests (besides role based `bulk_edit_rules_actions.cy.ts`) - `bulk_duplicate_rules.cy.ts` - `bulk_edit_rules.cy.ts` - `bulk_edit_rules_data_view.cy.ts` ## Flaky test runner `detection_response/rule_management/rule_actions/bulk_actions` folder [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3596) 🟢
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Oct 18, 2023
elastic#168307) **Addresses:** elastic#161540 ## Summary This PR unskips rules bulk actions Serverless Cypress tests (besides role based `bulk_edit_rules_actions.cy.ts`) - `bulk_duplicate_rules.cy.ts` - `bulk_edit_rules.cy.ts` - `bulk_edit_rules_data_view.cy.ts` ## Flaky test runner `detection_response/rule_management/rule_actions/bulk_actions` folder [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3596) 🟢 (cherry picked from commit 4f91082)
kibanamachine
referenced
this issue
Oct 18, 2023
…ss tests (#168307) (#169322) # Backport This will backport the following commits from `main` to `8.11`: - [[Security Solution] Unskip rules bulk actions Serverless Cypress tests (#168307)](#168307) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-10-18T22:26:55Z","message":"[Security Solution] Unskip rules bulk actions Serverless Cypress tests (#168307)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips rules bulk actions Serverless Cypress tests (besides role based `bulk_edit_rules_actions.cy.ts`)\r\n\r\n- `bulk_duplicate_rules.cy.ts`\r\n- `bulk_edit_rules.cy.ts`\r\n- `bulk_edit_rules_data_view.cy.ts`\r\n\r\n## Flaky test runner\r\n\r\n\r\n`detection_response/rule_management/rule_actions/bulk_actions` folder [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3596) 🟢","sha":"4f9108273c3bbdf03af26a8a0d2b7a571311751a","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","v8.11.0","v8.12.0"],"number":168307,"url":"https://github.com/elastic/kibana/pull/168307","mergeCommit":{"message":"[Security Solution] Unskip rules bulk actions Serverless Cypress tests (#168307)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips rules bulk actions Serverless Cypress tests (besides role based `bulk_edit_rules_actions.cy.ts`)\r\n\r\n- `bulk_duplicate_rules.cy.ts`\r\n- `bulk_edit_rules.cy.ts`\r\n- `bulk_edit_rules_data_view.cy.ts`\r\n\r\n## Flaky test runner\r\n\r\n\r\n`detection_response/rule_management/rule_actions/bulk_actions` folder [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3596) 🟢","sha":"4f9108273c3bbdf03af26a8a0d2b7a571311751a"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/168307","number":168307,"mergeCommit":{"message":"[Security Solution] Unskip rules bulk actions Serverless Cypress tests (#168307)\n\n**Addresses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips rules bulk actions Serverless Cypress tests (besides role based `bulk_edit_rules_actions.cy.ts`)\r\n\r\n- `bulk_duplicate_rules.cy.ts`\r\n- `bulk_edit_rules.cy.ts`\r\n- `bulk_edit_rules_data_view.cy.ts`\r\n\r\n## Flaky test runner\r\n\r\n\r\n`detection_response/rule_management/rule_actions/bulk_actions` folder [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3596) 🟢","sha":"4f9108273c3bbdf03af26a8a0d2b7a571311751a"}}]}] BACKPORT--> Co-authored-by: Maxim Palenov <[email protected]>
maximpn
added a commit
that referenced
this issue
Oct 19, 2023
…169243) **Addresses:** #161540 ## Summary This PR unskips `urls/not_found.cy.ts` Serverless Cypress tests. ## Flaky test runner `urls/not_found.cy.ts` [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3590)
maximpn
added a commit
that referenced
this issue
Oct 25, 2023
**Addreses:** #161540 ## Summary This PR unskips `search_bar.cy.ts` Serverless Cypress tests. ## Details Besides just unskipping `search_bar.cy.ts` this PR also makes sure the test isn't flaky by making `operator` required in `fillAddFilterForm()`. It turned out the test works only if the Cypress window is in focus when an operator isn't set. Such behavior can lead to test flakiness in CI. This way choosing an operator via keyboard is a safer option. ## Flaky test runner `search_bar.cy.ts` [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3614)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Oct 25, 2023
…ic#169347) **Addreses:** elastic#161540 ## Summary This PR unskips `search_bar.cy.ts` Serverless Cypress tests. ## Details Besides just unskipping `search_bar.cy.ts` this PR also makes sure the test isn't flaky by making `operator` required in `fillAddFilterForm()`. It turned out the test works only if the Cypress window is in focus when an operator isn't set. Such behavior can lead to test flakiness in CI. This way choosing an operator via keyboard is a safer option. ## Flaky test runner `search_bar.cy.ts` [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3614) (cherry picked from commit 906987c)
kibanamachine
referenced
this issue
Oct 26, 2023
…#169347) (#169765) # Backport This will backport the following commits from `main` to `8.11`: - [[Security Solituon] Unskip search bar Serverless Cypress tests (#169347)](#169347) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-10-25T09:24:32Z","message":"[Security Solituon] Unskip search bar Serverless Cypress tests (#169347)\n\n**Addreses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips `search_bar.cy.ts` Serverless Cypress tests.\r\n\r\n## Details\r\n\r\nBesides just unskipping `search_bar.cy.ts` this PR also makes sure the test isn't flaky by making `operator` required in `fillAddFilterForm()`. It turned out the test works only if the Cypress window is in focus when an operator isn't set. Such behavior can lead to test flakiness in CI. This way choosing an operator via keyboard is a safer option.\r\n\r\n## Flaky test runner\r\n\r\n`search_bar.cy.ts` [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3614)","sha":"906987c2860b53b91d449bc164957857adddc06a","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["test","release_note:skip","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","v8.11.0","v8.12.0"],"number":169347,"url":"https://github.com/elastic/kibana/pull/169347","mergeCommit":{"message":"[Security Solituon] Unskip search bar Serverless Cypress tests (#169347)\n\n**Addreses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips `search_bar.cy.ts` Serverless Cypress tests.\r\n\r\n## Details\r\n\r\nBesides just unskipping `search_bar.cy.ts` this PR also makes sure the test isn't flaky by making `operator` required in `fillAddFilterForm()`. It turned out the test works only if the Cypress window is in focus when an operator isn't set. Such behavior can lead to test flakiness in CI. This way choosing an operator via keyboard is a safer option.\r\n\r\n## Flaky test runner\r\n\r\n`search_bar.cy.ts` [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3614)","sha":"906987c2860b53b91d449bc164957857adddc06a"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/169347","number":169347,"mergeCommit":{"message":"[Security Solituon] Unskip search bar Serverless Cypress tests (#169347)\n\n**Addreses:** https://github.com/elastic/kibana/issues/161540\r\n\r\n## Summary\r\n\r\nThis PR unskips `search_bar.cy.ts` Serverless Cypress tests.\r\n\r\n## Details\r\n\r\nBesides just unskipping `search_bar.cy.ts` this PR also makes sure the test isn't flaky by making `operator` required in `fillAddFilterForm()`. It turned out the test works only if the Cypress window is in focus when an operator isn't set. Such behavior can lead to test flakiness in CI. This way choosing an operator via keyboard is a safer option.\r\n\r\n## Flaky test runner\r\n\r\n`search_bar.cy.ts` [150 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3614)","sha":"906987c2860b53b91d449bc164957857adddc06a"}}]}] BACKPORT--> Co-authored-by: Maxim Palenov <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Epic: #153633
Depends on: #161537, #164441
Summary
We need to create a suite of Cypress E2E tests that will run against Serverless environments.
In
x-pack/test/security_solution_cypress, we already have existing Cypress tests that run "in ESS", meaning against non-serverless environments. We need to start gradually including these Cypress tests in the Serverless suite by:BROKEN_IN_SERVERLESSand removing this tagSKIP_IN_SERVERLESSand removing this tagSERVERLESSIdeally, we'd do it gradually, in multiple PRs, starting from covering the most critical features and use cases, and proceeding with less important ones.
Todo
Tests that depend on roles are out of the scope.
The text was updated successfully, but these errors were encountered: