[Security Solution] Find a way to reuse tests between ESS and Serverless suites

[banderror]

Related PRs: #157166, #158707

Summary

Security Solution teams are going to need to start adding tests, that will be executed against Serverless environments, to the x-pack/test_serverless folder. In this folder, we will need to have both API integration and E2E functional tests. Support for Cypress was added in #158707.

We already have a lot of existing tests, for example:

• E2E functional tests: x-pack/plugins/security_solution/cypress
• API integration tests: x-pack/test/detection_engine_api_integration

This is a lot of existing tests, and we'd like some of them to run in both ESS and Serverless. Instead of duplicating them for Serverless, we'd like to be able to reuse them as much as possible. This is especially important for Detections features that are going to work mostly the same way in ESS and Serverless.

How we could do this

Some thoughts:

• It would be great if we could import reusable code from ESS tests into serverless tests.
• Reusable code could be whole tests, test "templates", or helper functions.
• For test "templates" we could use dependency injection to be able to "map" or "configure" them for ESS or Serverless

Todo

• Get feedback from AppEx-qa and other solution teams
• Make a POC of reusing E2E (Cypress) tests from x-pack/plugins/security_solution/cypress
  • [Security Solution] Reusing Cypress tests POC1 #162023
  • [Security Solution] Reusing Cypress tests POC2 #162058
  • [Security Solution][Serverless] Reusing Cypress tests for Serverless infrastructure #162698
• Make a POC of reusing API integration (FTR) tests from x-pack/test/detection_engine_api_integration
  • [Security Solution][Detection Engine] Running API tests in Serverless & ESS using Mocha Tagging #166755

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

[kevinlog]

Adding Defend Workflows label so we can follow along and give feedback on this

[MadameSheema]

Sharing some information to give more context to the ticket:

There is a Kibana Serverless testing document with some guidelines around how to proceed with the automation on serverless, agreed by various stakeholders.

About the structure, the original guideline is the following one:

The current structure of tests for the stateful / “base” Kibana will not be changed. These tests will continue to cover functionality for the current offering.

Tests for serverless projects will be clearly separated into directories corresponding to the respective project. These tests should cover functionality specific to the serverless project and how it affects “base” Kibana (e.g. different menu structure, added/removed APIs, changes to how components are included, changes to user workflows, …). This also includes areas where Kibana depends on ES functionality that is changed for serverless.

If required, base tests may be duplicated into project tests and then modified to the project’s needs. But there won’t be an import / reuse of existing tests* as that would defeat the purpose of a clear structure, could lead to code branches in test files and would cause higher maintenance cost.

* Re-use of common code like helper scripts, page objects and test services is allowed, but we need separate test files.

Test coverage for a serverless project will then be given through the combination of “base” Kibana tests (running against stateful Kibana on stateful ES) and the respective serverless project tests (running against that serverless Kibana project on serverless ES).

For the Cypress-specific side of things, that would be translated to the reuse of tasks, screens, and fixtures. Meanwhile we don't migrate the stateful solution to datastreams, I don't know if we could be able to reuse archives as well.

[kevinlog]

Notes from a @elastic/security-defend-workflows meeting:

1. We also maintain FTR tests. We are writing new tests in Cypress, but still want to support
   a. Short term: For FTR tests could probably copy the config in the Serverless plugin
   b. Medium term: Port FTR tests to Cypress
2. We agree that we should try to reuse tests in ESS with Serverless
3. @gergoabraham from our team is going be the point person from Defend Workflows
4. We have general concerns over testing against a true Serverless env vs. the simulated one that would be included in CI. Out of scope for this effort, but a good question.

[MadameSheema]

@kevinlog may you please elaborate on this?

4. We have general concerns over testing against a true Serverless env vs. the simulated one that would be included in CI. Out of scope for this effort, but a good question.

At some point in the pipeline/flow we'll use a real serverless environment so it would be great to hear about those concerns to be taken into consideration with time :)

[kevinlog]

@MadameSheema

At some point in the pipeline/flow we'll use a real serverless environment so it would be great to hear about those concerns to be taken into consideration with time :)

Yes, this was our concern. It's understood that this would be a longer term goal. If we have a plan to use a real serverless environment then that works!

[gergoabraham]

Sharing some information to give more context to the ticket:

There is a Kibana Serverless testing document with some guidelines around how to proceed with the automation on serverless, agreed by various stakeholders.

Test coverage for a serverless project will then be given through the combination of “base” Kibana tests (running against stateful Kibana on stateful ES) and the respective serverless project tests (running against that serverless Kibana project on serverless ES).

I'm sorry if I miss something, since I'm just familiarising myself with the serverless offering and its testing plan, but based on the doc @MadameSheema shared, it is not a global goal to run the same tests against both stateful and serverless Kibana, although, I'd imagine that would help a lot to ensure SW quality (but results far slower test runs).

@banderror could you please elaborate what the goal is here? is it to opt in some existing ESS tests to ensure features work on serverless? or maybe all of them?

we'd like some of them [existing tests] to run in both ESS and Serverless

[banderror]

@gergoabraham I'm sorry for the terribly delayed response.

based on the doc @MadameSheema shared, it is not a global goal to run the same tests against both stateful and serverless Kibana

True, based on the information I have I think it's not a global (Kibana-wide) goal. The Kibana Serverless e2e Test Guide (internal document) proposes that:

• Overall test coverage for serverless Kibana is given through the combination of stateful tests and serverless tests.
• Stateful tests are giving us base confidence in the functionality of a shared feature. But they can not run against a serverless deployment.
• So beyond that, teams have to assess if and how a feature is used in serverless projects and determine required test coverage.
• Over time, we expect the majority of features to be covered by serverless tests.

We've also got the following comment from the AppEx QA team:

We understand that there's a desire to reuse some test code across stateful and serverless. But there are also good reasons to not do it. There's a very similar thread in this channel, it's starting here: internal slack link
for FTR tests, we're looking into where to add a place for test that can run the same on stateful and serverless. But many tests actually have some expectations about the deployment 🤷

Several teams have expressed some concerns with that approach, including the Threat Hunting folks. This especially applies to the two teams under Detection and Response, because most of our functionality is going to work the same way in ESS (self-managed) and Serverless. All the API endpoints available in Serverless are going to work exactly the same way as in ESS too. So we want to be able to reuse our existing test coverage, instead of duplicating it in a different folder. We also don't think that not covering Serverless with existing tests and only writing Serverless tests for features that work differently in Serverless, would be a good way for developing the Detection Engine and maybe other features of Security Solution. Many people from D&R and AET are on board with the idea of reusing existing tests, including @oatkiller @marshallmain @MadameSheema @MindyRS.

@banderror could you please elaborate what the goal is here? is it to opt in some existing ESS tests to ensure features work on serverless? or maybe all of them?

The goal is to reuse our existing E2E (Cypress) and API integration (FTR) tests in all cases where a feature under test works exactly the same way in Serverless and ESS. For those features that work differently, we will need to have ESS-specific and Serverless-specific tests. The amount of reusable and environment-specific tests will depend on every area and team, but for D&R features and tests we expect a very high ratio of reusable-to-specific tests. When we figure out how to reuse tests, we can start to gradually opt them in (add them to the serverless suite) until we feel that the coverage for Serverless is full or sufficient.

More about reusing Cypress tests in the next comment.

@gergoabraham Please let me know if I was able to answer your questions.

[banderror]

@MadameSheema has been working on POCs for reusing our existing x-pack/plugins/security_solution/cypress tests in x-pack/test_serverless:

• [Security Solution] Reusing Cypress tests POC1 #162023
• [Security Solution] Reusing Cypress tests POC2 #162058

I guess people tend to prefer the second POC that allows us to keep all Cypress tests (reusable, serverless-specific, and ess-specific) in x-pack/plugins/security_solution/cypress and differentiate between them via tagging.

@MadameSheema started to work on the implementation PR based on the second POC: #162698

cc @gergoabraham

[gergoabraham]

@banderror Thank you for your detailed answer, now I think I have a good grasp of the context. There are some really good arguments on both sides of reusing or not reusing tests, but as our features from Defend Workflows team work practically the same on both environments, I totally agree with the reuse of the tests. Also, I support the idea of tagging: I think we rather implement new functionalities organized by features and not by environments, so it's much easier to overview a feature and its tests in one place, and making sure it's okay for serverless only by adding a tag.

[banderror]

The problem of reusing API integration (FTR) tests has been addressed by @WafaaNasr in #166755.

The leftover work which is out of the scope of this ticket:

• Moving D&R integration tests from x-pack/test/detection_engine_api_integration to the new folder x-pack/test/security_solution_api_integration and enabling them in Serverless in the PR pipeline. The new folder can be used by other teams as well.
• Enabling API integration tests in MKI pipelines (for the QA, Staging, Prod environments)
• Enabling Cypress tests in MKI pipelines (for the QA, Staging, Prod environments)