Failing test: Security Solution Cypress.x-pack/test/security_solution_cypress/cypress/e2e/detection_response/detection_engine/detection_alerts/assignments/assignments_serverless_complete·cy·ts - Alert user assignment - Serverless Complete Authorization / RBAC users with editing privileges should be able to update assignees users with editing privileges should be able to update assignees #172557

kibanamachine · 2023-12-05T11:19:17Z

A test failed on a tracked branch

CypressError: Timed out retrying after 150000ms: The command was expected to run against origin `http://localhost:5638` but the application is at origin `https://auth.elastic.co`.

This commonly happens when you have either not navigated to the expected origin or have navigated away unexpectedly.

Using `cy.origin()` to wrap the commands run on `https://auth.elastic.co` will likely fix this issue.

`cy.origin('https://auth.elastic.co', () => {`
`  <commands targeting https://auth.elastic.co go here>`
`})`

https://on.cypress.io/cy-visit-succeeded-but-commands-fail
    at cypressErr (http://localhost:5638/__cypress/runner/cypress_runner.js:75008:18)
    at Object.errByPath (http://localhost:5638/__cypress/runner/cypress_runner.js:75063:10)
    at Object.commandCanCommunicateWithAUT (http://localhost:5638/__cypress/runner/cypress_runner.js:145193:90)
    at <unknown> (http://localhost:5638/__cypress/runner/cypress_runner.js:132145:22)
    at Object.subjectFn (http://localhost:5638/__cypress/runner/cypress_runner.js:144030:16)
    at $Cy.verifyUpcomingAssertions (http://localhost:5638/__cypress/runner/cypress_runner.js:143374:31)
    at onRetry (http://localhost:5638/__cypress/runner/cypress_runner.js:144021:15)
    at tryCatcher (http://localhost:5638/__cypress/runner/cypress_runner.js:1807:23)
    at Promise.attempt.Promise.try (http://localhost:5638/__cypress/runner/cypress_runner.js:4315:29)
    at whenStable (http://localhost:5638/__cypress/runner/cypress_runner.js:143908:68)
    at <unknown> (http://localhost:5638/__cypress/runner/cypress_runner.js:143849:14)
    at tryCatcher (http://localhost:5638/__cypress/runner/cypress_runner.js:1807:23)
    at Promise._settlePromiseFromHandler (http://localhost:5638/__cypress/runner/cypress_runner.js:1519:31)
    at Promise._settlePromise (http://localhost:5638/__cypress/runner/cypress_runner.js:1576:18)
    at Promise._settlePromise0 (http://localhost:5638/__cypress/runner/cypress_runner.js:1621:10)
    at Promise._settlePromises (http://localhost:5638/__cypress/runner/cypress_runner.js:1701:18)
    at Promise._fulfill (http://localhost:5638/__cypress/runner/cypress_runner.js:1645:18)
    at <unknown> (http://localhost:5638/__cypress/runner/cypress_runner.js:5450:46)
From Your Spec Code:
    at waitForPageTitleToBeShown (webpack:///./tasks/alert_assignments.ts:50:21)
    at loadPageAs (webpack:///./tasks/alert_assignments.ts:56:2)
    at eval (webpack:///./e2e/detection_response/detection_engine/detection_alerts/assignments/assignments_serverless_complete.cy.ts:72:21)
at Array.forEach (<anonymous>)
    at Context.eval (webpack:///./e2e/detection_response/detection_engine/detection_alerts/assignments/assignments_serverless_complete.cy.ts:71:16)

First failure: CI Build - main

The text was updated successfully, but these errors were encountered:

kibanamachine · 2023-12-05T16:59:34Z

New failure: CI Build - main

mistic · 2023-12-05T17:41:41Z

Skipped.

main: 6de3768

elasticmachine · 2023-12-15T21:24:37Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2023-12-15T21:24:37Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

- elastic#173429 - elastic#172520 - elastic#172557

@MadameSheema

## Summary These changes fix flaky alert assignments cypress tests: - #173429 - #172520 - #172557 @MadameSheema mentioned that there were a lot of changes around login functionality recently which could have fixed original issues. We agreed that we will run a flaky test runner and if everything works fine we would un-skip tests and monitor them after the merge. ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ESS 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5064) - [Serverless 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5052)

e40pud · 2024-02-07T10:32:59Z

We enabled these tests in #175965 and will be monitoring for new flakes.

@MadameSheema

## Summary These changes fix flaky alert assignments cypress tests: - elastic#173429 - elastic#172520 - elastic#172557 @MadameSheema mentioned that there were a lot of changes around login functionality recently which could have fixed original issues. We agreed that we will run a flaky test runner and if everything works fine we would un-skip tests and monitor them after the merge. ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ESS 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5064) - [Serverless 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5052)

@MadameSheema

## Summary These changes fix flaky alert assignments cypress tests: - elastic#173429 - elastic#172520 - elastic#172557 @MadameSheema mentioned that there were a lot of changes around login functionality recently which could have fixed original issues. We agreed that we will run a flaky test runner and if everything works fine we would un-skip tests and monitor them after the merge. ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ESS 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5064) - [Serverless 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5052)

kibanamachine · 2024-02-22T11:15:28Z

New failure: CI Build - 8.13

yctercero · 2024-02-29T13:14:27Z

@e40pud can we review this test? Are there ways we can mock some of the ES calls we don't need to test explicitly?

@MadameSheema

## Summary These changes fix flaky alert assignments cypress tests: - elastic#173429 - elastic#172520 - elastic#172557 @MadameSheema mentioned that there were a lot of changes around login functionality recently which could have fixed original issues. We agreed that we will run a flaky test runner and if everything works fine we would un-skip tests and monitor them after the merge. ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ESS 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5064) - [Serverless 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5052)

@yctercero

## Summary Addresses: * #176529 * #172557 * #177573 * #173429 Fix flaky alert assignments tests. I split assignments tests into two groups: tests with one assignee available and tests with multiple assignees. Right now there is a flakiness in tests with multiple assignees. Most probably it is happening because we do multiple login calls in a row to make sure we activate different users to make them available for assignments: ``` // Login into accounts so that they got activated and visible in user profiles list login(ROLES.t1_analyst); login(ROLES.t2_analyst); login(ROLES.t3_analyst); login(ROLES.soc_manager); login(ROLES.detections_admin); login(ROLES.platform_engineer); ``` These tests are tend to be flaky and it is possible that kibana operations team will skip those. To make sure that we run basic cypress verification of alert assignments feature we decided to add tests with only one assignee available (current user) which allows us to avoid multiple consecutive login calls. Also, as part of these changes I removed unnecessary logins and un-skipped #176529 ## NOTE After discussing these failure with the team, we decided to remove tests which are covered by the integration and unit tests. While fixing the flakiness we realised that we do unnecessary work trying to fight the internal errors within elastic search on serverless when we do multiple user logins in a row. Instead we will rely on: * integration tests coverage of API related functionality including RBAC * unit tests coverage of all assignments UI components * cypress tests coverage of basic UI interaction with the alert assignments with only one user available for the assignments cc @yctercero ### Checklist Delete any items that are not applicable to this PR. - [x] [Flaky Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was used on any tests changed - [ESS 50 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5181) - [Serverless 97 times](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/5182) --------- Co-authored-by: Kibana Machine <[email protected]>

e40pud · 2024-03-06T11:16:13Z

We removed these tests in #176930 in favour of combination of basic e2e and extensive integration + unit tests.

kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Dec 5, 2023

botelastic bot added the needs-team Issues missing a team label label Dec 5, 2023

mistic added the Team:Detection Engine Security Solution Detection Engine Area label Dec 5, 2023

botelastic bot removed the needs-team Issues missing a team label label Dec 5, 2023

mistic added blocker skipped-test v8.12.0 labels Dec 5, 2023

mistic added a commit that referenced this issue Dec 5, 2023

skip flaky suite (#172557)

6de3768

banderror added triage_needed Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. labels Dec 15, 2023

banderror assigned yctercero Dec 15, 2023

yctercero assigned e40pud and unassigned yctercero Jan 2, 2024

yctercero added v8.12.1 v8.13.0 labels Jan 8, 2024

yctercero mentioned this issue Jan 11, 2024

[Tests][Detection Engine] Track status of FTR/Cypress tests for detection engine #171366

Open

e40pud added a commit to e40pud/kibana that referenced this issue Jan 31, 2024

Fix flaky tests

b6f3ad4

- elastic#173429 - elastic#172520 - elastic#172557

e40pud mentioned this issue Jan 31, 2024

Fix flaky tests #175965

Merged

1 task

e40pud closed this as completed Feb 7, 2024

kibanamachine removed v8.12.0 blocker v8.12.1 skipped-test labels Feb 7, 2024

kibanamachine removed the v8.13.0 label Feb 7, 2024

kibanamachine reopened this Feb 22, 2024

e40pud mentioned this issue Mar 5, 2024

Flaky alert assignment tests #176930

Merged

1 task

e40pud closed this as completed Mar 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kibanamachine commented Dec 5, 2023 •

edited

Loading

kibanamachine commented Dec 5, 2023

mistic commented Dec 5, 2023

elasticmachine commented Dec 15, 2023

elasticmachine commented Dec 15, 2023

e40pud commented Feb 7, 2024

kibanamachine commented Feb 22, 2024

yctercero commented Feb 29, 2024

e40pud commented Mar 6, 2024

Comments

kibanamachine commented Dec 5, 2023 • edited Loading

kibanamachine commented Dec 5, 2023

mistic commented Dec 5, 2023

elasticmachine commented Dec 15, 2023

elasticmachine commented Dec 15, 2023

e40pud commented Feb 7, 2024

kibanamachine commented Feb 22, 2024

yctercero commented Feb 29, 2024

e40pud commented Mar 6, 2024

kibanamachine commented Dec 5, 2023 •

edited

Loading