[Security Solution] Stabilize Cypress

[MadameSheema]

Kibana Operations team is open to have our tests executed on every PR, before arriving to that point there are a set of criteria our tests needs to meet first.

The aim of this ticket is to centralize all the work we have to do first before enabling our tests:

• To have a clear picture of the ownership of the tests
• Remove flakiness
  • [Security Solution] Fix flaky tests: common tasks #161505
  • [Security Solution] Fix flaky tests: Detection Engine area #161506
  • [Security Solution] Fix flaky tests: Rule Management area #161507
  • [Security Solution] Explore, fix flakey Cypress tests #161874
  • https://github.com/elastic/security-team/issues/7123
  • Remove Threat Hunting Flakiness
• Improve test execution (each parallel task should be executed in less than 40min)
  • Improve Detections and Response execution time
  • Improve Threat Hunting execution time
• We can take advantage of the work done in Improve reliability and performance of Cypress tests @patrykkopycinski

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[spong]

Thanks for coordinating this effort and pushing it forward @MadameSheema! Really excited this is so close happening finally! 🙂

Two questions with regards to the acceptance criteria from the ops side:

1. In the event of flake, what automation exists for auto-skipping and follow-up issue creation/assignment? Can we get the @kibanamachine automation (sample jest issue) working with the cypress tests too? Is there a buildkite dashboard tracking failures across all builds we can reference?
2. Even with documenting best practices for test development and to reduce flake (#153662 #153655), flake will find a way 🦖 🦕. Would it be possible to get cypress support in the flakey test runner? Then we can at least stress test new suites before hitting main.

edit: Looks like the flakey test runner supports the cypress CI jobs now! I hadn't realized, so we should be good here now 🙂

[MadameSheema]

Hey @spong!! Yes! We should work on the first point as well, there are even more tasks we will need to implement, like automate the number of parallel tasks that are executed and so on.... the above is just the first set of things we need to do :)

[oatkiller]

I believe this is a duplicate of https://github.com/elastic/security-team/issues/6769

[oatkiller]

@MadameSheema Are you OK to close this in favor of https://github.com/elastic/security-team/issues/6769 ?

[MadameSheema]

Hey @oatkiller I believe there are some tasks/efforts here that are not tracked in the shared ticket, can we include this ticket as part of the tasks? The first agreement I had with Kibana operations about having our tests enabled in every PR was to first stabilize Cypress and the first milestones were explained above.

[VidhiRambhia]

Hi Team,

I am interested in addressing these open items. Is 'Improve test execution (each parallel task should be executed in less than 40min)' actively being worked on?

[MadameSheema]

Closing in favour of: #173508