Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution][Endpoint] Unskip flaky tests (document signing & response console - isolate) #169539

Merged

Conversation

ashokaditya
Copy link
Member

@ashokaditya ashokaditya commented Oct 23, 2023

Summary

Split test to avoid long running times.

Fixes
#168296
#168390

Flaky test runner

Checklist

@ashokaditya ashokaditya added release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution OLM Sprint v8.12.0 labels Oct 23, 2023
@ashokaditya ashokaditya self-assigned this Oct 23, 2023
@ashokaditya ashokaditya marked this pull request as ready for review October 26, 2023 12:53
@ashokaditya ashokaditya requested a review from a team as a code owner October 26, 2023 12:53
@ashokaditya ashokaditya requested review from pzl and dasansol92 October 26, 2023 12:53
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

@ashokaditya ashokaditya changed the title [Security Solution][Endpoint] Unskip document signing flaky test [Security Solution][Endpoint] Unskip flaky tests (document signing & response console - isolate) Oct 30, 2023
ashokaditya and others added 4 commits October 31, 2023 10:10
The test was flaky as it depends on the previous `isolate` test to isolate the host. If it doesn't then the release test can not assert that the host is isolated to start with when doing to `release` test.

This makes the test independent from the previous isolate.
@ashokaditya ashokaditya force-pushed the task/unskip-document-signing-flaky-test branch from 14270a6 to 473a98a Compare November 1, 2023 08:42
@ashokaditya ashokaditya force-pushed the task/unskip-document-signing-flaky-test branch from a9babde to 3d56416 Compare November 1, 2023 09:09
@kibana-ci
Copy link
Collaborator

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @ashokaditya

@ashokaditya ashokaditya merged commit a3d71f2 into elastic:main Nov 1, 2023
@ashokaditya ashokaditya deleted the task/unskip-document-signing-flaky-test branch November 1, 2023 20:33
@kibanamachine
Copy link
Contributor

💔 All backports failed

Status Branch Result
8.11 Backport failed because of merge conflicts

You might need to backport the following PRs to 8.11:
- [Security Solution][Analyzer] - Fix analyzer panel text (#170329)
- [RAM] Fix flaky test get_global_execution_kpi (#170328)

Manual backport

To create the backport manually run:

node scripts/backport --pr 169539

Questions ?

Please refer to the Backport tool documentation

@ashokaditya
Copy link
Member Author

💚 All backports created successfully

Status Branch Result
8.11

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

ashokaditya added a commit to ashokaditya/kibana that referenced this pull request Nov 2, 2023
…response console - isolate) (elastic#169539)

## Summary

Split test to avoid long running times.

Fixes
elastic#168296
elastic#168390

**Flaky test runner**
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3754
x 150 (3 failures for document signing: 7, 101, 141)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3809
x 50 (all pass for document_signing.cy.ts)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3835
x 50 (2 fails for document signing and 1 fail for response console
release due to timeout on response delay)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3842
x 25 (all pass for document_singing.cy.ts and
response_console_actions.cy.ts)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3843
x 25 (3 fails for document_signing.cy.ts related to timeout for action
response)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3844
x 100 (all pass for release, 3 failures for document signing due to time
out waiting for action response)

-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3848
x 50 ( all pass, 1 unrelated failure related to
`endpoints_list_response_console.cy·ts`)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3849
x 50 (all pass)
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

(cherry picked from commit a3d71f2)

# Conflicts:
#	x-pack/plugins/security_solution/public/management/cypress/e2e/response_actions/response_console_actions.cy.ts
ashokaditya added a commit that referenced this pull request Nov 2, 2023
…ning & response console - isolate) (#169539) (#170381)

# Backport

This will backport the following commits from `main` to `8.11`:
- [[Security Solution][Endpoint] Unskip flaky tests (document signing &
response console - isolate)
(#169539)](#169539)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT
[{"author":{"name":"Ash","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-11-01T20:33:36Z","message":"[Security
Solution][Endpoint] Unskip flaky tests (document signing & response
console - isolate) (#169539)\n\n## Summary\r\n\r\nSplit test to avoid
long running
times.\r\n\r\nFixes\r\nhttps://github.com//issues/168296\r\nhttps://github.com//issues/168390\r\n\r\n**Flaky
test
runner**\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3754\r\nx
150 (3 failures for document signing: 7, 101,
141)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3809\r\nx
50 (all pass for
document_signing.cy.ts)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3835\r\nx
50 (2 fails for document signing and 1 fail for response
console\r\nrelease due to timeout on response
delay)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3842\r\nx
25 (all pass for document_singing.cy.ts
and\r\nresponse_console_actions.cy.ts)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3843\r\nx
25 (3 fails for document_signing.cy.ts related to timeout for
action\r\nresponse)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3844\r\nx
100 (all pass for release, 3 failures for document signing due to
time\r\nout waiting for action
response)\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3848\r\nx
50 ( all pass, 1 unrelated failure related
to\r\n`endpoints_list_response_console.cy·ts`)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3849\r\nx
50 (all pass)\r\n### Checklist\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"a3d71f2534a56dd070982d6d8c60e56aa43ae2fa","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team:Defend
Workflows","OLM
Sprint","v8.11.0","v8.12.0"],"number":169539,"url":"https://github.com/elastic/kibana/pull/169539","mergeCommit":{"message":"[Security
Solution][Endpoint] Unskip flaky tests (document signing & response
console - isolate) (#169539)\n\n## Summary\r\n\r\nSplit test to avoid
long running
times.\r\n\r\nFixes\r\nhttps://github.com//issues/168296\r\nhttps://github.com//issues/168390\r\n\r\n**Flaky
test
runner**\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3754\r\nx
150 (3 failures for document signing: 7, 101,
141)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3809\r\nx
50 (all pass for
document_signing.cy.ts)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3835\r\nx
50 (2 fails for document signing and 1 fail for response
console\r\nrelease due to timeout on response
delay)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3842\r\nx
25 (all pass for document_singing.cy.ts
and\r\nresponse_console_actions.cy.ts)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3843\r\nx
25 (3 fails for document_signing.cy.ts related to timeout for
action\r\nresponse)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3844\r\nx
100 (all pass for release, 3 failures for document signing due to
time\r\nout waiting for action
response)\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3848\r\nx
50 ( all pass, 1 unrelated failure related
to\r\n`endpoints_list_response_console.cy·ts`)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3849\r\nx
50 (all pass)\r\n### Checklist\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"a3d71f2534a56dd070982d6d8c60e56aa43ae2fa"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/169539","number":169539,"mergeCommit":{"message":"[Security
Solution][Endpoint] Unskip flaky tests (document signing & response
console - isolate) (#169539)\n\n## Summary\r\n\r\nSplit test to avoid
long running
times.\r\n\r\nFixes\r\nhttps://github.com//issues/168296\r\nhttps://github.com//issues/168390\r\n\r\n**Flaky
test
runner**\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3754\r\nx
150 (3 failures for document signing: 7, 101,
141)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3809\r\nx
50 (all pass for
document_signing.cy.ts)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3835\r\nx
50 (2 fails for document signing and 1 fail for response
console\r\nrelease due to timeout on response
delay)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3842\r\nx
25 (all pass for document_singing.cy.ts
and\r\nresponse_console_actions.cy.ts)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3843\r\nx
25 (3 fails for document_signing.cy.ts related to timeout for
action\r\nresponse)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3844\r\nx
100 (all pass for release, 3 failures for document signing due to
time\r\nout waiting for action
response)\r\n\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3848\r\nx
50 ( all pass, 1 unrelated failure related
to\r\n`endpoints_list_response_console.cy·ts`)\r\n-\r\nhttps://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3849\r\nx
50 (all pass)\r\n### Checklist\r\n- [x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"a3d71f2534a56dd070982d6d8c60e56aa43ae2fa"}}]}]
BACKPORT-->
delanni pushed a commit to delanni/kibana that referenced this pull request Nov 6, 2023
…response console - isolate) (elastic#169539)

## Summary

Split test to avoid long running times.

Fixes
elastic#168296
elastic#168390

**Flaky test runner**
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3754
x 150 (3 failures for document signing: 7, 101, 141)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3809
x 50 (all pass for document_signing.cy.ts)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3835
x 50 (2 fails for document signing and 1 fail for response console
release due to timeout on response delay)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3842
x 25 (all pass for document_singing.cy.ts and
response_console_actions.cy.ts)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3843
x 25 (3 fails for document_signing.cy.ts related to timeout for action
response)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3844
x 100 (all pass for release, 3 failures for document signing due to time
out waiting for action response)

-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3848
x 50 ( all pass, 1 unrelated failure related to
`endpoints_list_response_console.cy·ts`)
-
https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/3849
x 50 (all pass)
### Checklist
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
OLM Sprint release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v8.11.0 v8.12.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants