[Security Solution] [Cypress] SAML for Serverless login and role testing #172655

MadameSheema · 2023-12-06T09:59:14Z

Relates to:

Summary

In this PR we are using the code implemented on #170417 and #172678 to allow SAML and role testing inside Cypress.

We are creating a Cypress task to use the above-developed code and be able to retrieve a session cookie given a role.
We updated the login task to know how we should perform the login depending if we are in Serverless (MKI or serverless FTR) or ESS
In the parallel serverless script:
- We are updating the BASE_ENV_URL variable to use the proper QA environment (pending to be done in follow-up PRs, to extract this value so it is not hardcoded cc @dkirchan )
- We are adding the IS_SERVERLESS environment variable needed for the logic on the login task. This changed implied to update the es_archiver file to continue work as expected.
- We have added the TEST_CLOUD_HOST_NAME environment variable needed for the code we are reusing to retrieve the session cookie for MKI.
We have updated the Security Solution quality gate script to set the role_users.json file needed by the code we are reusing to get the different session cookies on MKI
We have adjusted the tests because the username now follows the pattern test <role> (@dmlemeshko is it possible to have as username just the role? Is this something that can impact other tests and teams?)
We have skipped a test that got unstable after the changes.

How to test it in your machine

Serverless FTR

Navigate to x-pack/test/security_solution_cypress
Execute yarn cypress:open:serverless
Click on E2E testing
Click on any test to execute it

Serverless MKI

Setup a valid Elastic Cloud API key for QA environment:

Navigate to QA environment.
Click on the User menu button located on the top right of the header.
Click on Organization.
Click on the API keys tab.
Click on Create API key button.
Add a name, set an expiration date, assign an organization owner role.
Click on Create API key
Save the value of the key

Store the saved key on ~/.elastic/cloud.json using the following format:

{
  "api_key": {
    "qa": "<API_KEY>"
  }
}

Store the email and password of the account you used to login in the QA Environment at the root directory of your Kibana project on .ftr/role_users.json, using the following format:

{
  "admin": {
    "email": "<email>",
    "password": "<password>"
  }
}

If you want to execute a test with a role different from the default one, make sure you have created the user under your organization and is added to the above json following the format:

{
  "admin": {
    "email": "<email>",
    "password": "<password>"
  },
  "<roleName>": {
    "email": "<email>",
    "password": "<password>"
  }
}

Navigate to x-pack/test/security_solution_cypress
Execute yarn cypress:open:qa:serverless
Click on E2E testing
Click on any test to execute it

…fix'

pheyos

As discussed in slack, we should try to use some shared functionality to handle all this SAML stuff. This will make it more stable and easier to maintain in the future. There's currently work going on to move the SAML test logic into the kbn-test package so it can also be used by Cypress tests, see #172678.
@MadameSheema is there a reason to NOT use the shared functionality?

MadameSheema · 2023-12-06T18:40:35Z

As discussed in slack, we should try to use some shared functionality to handle all this SAML stuff. This will make it more stable and easier to maintain in the future. There's currently work going on to move the SAML test logic into the kbn-test package so it can also be used by Cypress tests, see #172678.
@MadameSheema is there a reason to NOT use the shared functionality?

Not at all :) In this draft PR I'm reusing the initial code Dima wrote, I was not aware of the kbn-test package effort when I opened this draft.

I already synced with @dmlemeshko and will wait for the above-mentioned PR to finalize the integration with Cypress.

MadameSheema · 2023-12-11T13:58:31Z

/ci

…fix'

MadameSheema · 2023-12-11T14:39:32Z

/ci

MadameSheema · 2023-12-11T19:11:22Z

/ci

MadameSheema · 2023-12-12T10:12:06Z

/ci

MadameSheema · 2023-12-12T14:15:13Z

/ci

elasticmachine · 2023-12-12T16:05:22Z

Pinging @elastic/security-solution (Team: SecuritySolution)

MadameSheema · 2023-12-12T16:11:44Z

/ci

x-pack/test/security_solution_cypress/cypress/e2e/explore/overview/overview.cy.ts

charlie-pichette

Looks good. 👍🏼

dkirchan · 2023-12-12T16:41:55Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts

@@ -571,6 +571,7 @@ ${JSON.stringify(cypressConfigFile, null, 2)}
              KIBANA_PASSWORD: credentials.password,

              CLOUD_SERVERLESS: true,
+              IS_SERVERLESS: true,


Cant we use only one of these two flags in lines 573 and 574? If this is not a big rework.

These flags represent two different things.

IS_SERVERLESS is used for both FTR and cloud environments.

CLOUD_SERVERLESS is used for just deployed projects on the cloud, this cannot be used for the FTR serverless environment.

stephmilovic

LGTM for explore changes, thanks!

kibana-ci · 2023-12-12T19:01:41Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: 4dd7911

Failed CI Steps

Test Failures

[job] [logs] FTR Configs #29 / discover Discover Saved Queries Manage saved queries updates a saved query
[job] [logs] Serverless Rule Management - Security Solution Cypress Tests #2 / Related integrations related Integrations Advanced Setting is disabled rules management table should not display a badge with the installed integrations should not display a badge with the installed integrations

Metrics [docs]

✅ unchanged

History

💔 Build #183213 failed 618ad00
💔 Build #183152 failed 32d9b04
💛 Build #183096 was flaky 3b488d4
💛 Build #183043 was flaky 191ad94
💔 Build #182959 failed 9eaaa29

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @MadameSheema

delanni · 2023-12-13T10:29:55Z

...es/security_solution_quality_gate/security_solution_cypress/mki_security_solution_cypress.sh

@@ -16,6 +16,9 @@ export JOB=kibana-security-solution-chrome

 buildkite-agent meta-data set "${BUILDKITE_JOB_ID}_is_test_execution_step" "true"

+mkdir .ftr
+retry 5 5 vault kv get -format=json -field=data secret/kibana-issues/dev/security-quality-gate/role-users > .ftr/role_users.json


We recently implemented a change where we're accessing vault through a variable path. I think for this to be future-proof to some degree, we should also care for that branching in here. However, the utility function that we use cannot be parameterized with extra args in its current state.

If this is urgent, we can go with this.
If you have some time to chisel the edges, I'll follow up with the details, so you can add it to this PR.

@dkirchan can you please take a look at @delanni comment and estimate how long is going to take to make the changes? Thanks!

@delanni as discussed private, lets unblock this PR and I will add these changes as part of my PR that I am preparing: #173005

delanni

approved with suggested fix incoming in #173005

vitaliidm

Detection Response area LGTM. Thanks for this PR implementation

Successfully log in by different roles, created for test qa depoloyment

Note: in PR description , when testing MKI deployment, script in second step should be

Execute yarn cypress:open:qa:serverless

Observation: most of the tests against MKI failed when I run it locally, Chrome just crashed

We detected that the Chrome Renderer process just crashed.

We have failed the current spec but will continue running the next spec.

This can happen for a number of different reasons.

If you're running lots of tests on a memory intense application.
  - Try increasing the CPU/memory on the machine you're running on.
  - Try enabling experimentalMemoryManagement in your config file.
  - Try lowering numTestsKeptInMemory in your config file during 'cypress open'.

You can learn more here:

https://on.cypress.io/renderer-process-crashed

Have you seen it before? Is it possibly related to this PR?

vitaliidm · 2023-12-13T15:30:14Z

x-pack/test/security_solution_cypress/cypress/README.md

+}
+```
+
+As role names please use:


should list of these role match ROLES enum?

This is something I want to double check with @maximpn when he is back. The listed roles are the ones available on serverless and are the ones we can retrieve cookies from so I've the feeling we might be missing roles on that enum.

MadameSheema · 2023-12-13T15:48:11Z

@vitaliidm you are right around the script!! thanks for noticing! about the failure, I didn't see it before.

MadameSheema and others added 3 commits December 6, 2023 10:58

integrating SAML authentication in Cypress

8fda395

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

75001bc

…fix'

adding missing configuration

4183b4d

pheyos reviewed Dec 6, 2023

View reviewed changes

MadameSheema added 3 commits December 11, 2023 10:46

fixes typo

8097ab8

Merge branch 'main' into saml-cypress-integration

f6005f6

refactors to use the new available changes

4786b98

kibanamachine and others added 2 commits December 11, 2023 14:05

[CI] Auto-commit changed files from 'node scripts/lint_ts_projects --…

649094b

…fix'

refactors configuration

9eaaa29

MadameSheema added 2 commits December 11, 2023 17:29

Merge branch 'main' into saml-cypress-integration

19c88d5

adjusting tests

191ad94

MadameSheema added 2 commits December 12, 2023 11:10

adjustments

498e8b9

Merge branch 'main' into saml-cypress-integration

3b488d4

MadameSheema added 2 commits December 12, 2023 15:14

updating readme

5215f31

Merge branch 'main' into saml-cypress-integration

32d9b04

MadameSheema changed the title ~~integrating SAML authentication in Cypress~~ [Security Solution] [Cypress] SAML for Serverless login and role testing Dec 12, 2023

MadameSheema mentioned this pull request Dec 12, 2023

[SERVERLESS] Skipped test: x-pack/test/security_solution_cypress/cypress/e2e/explore/overview/overview.cy.ts Favorite Timelines #173168

Open

MadameSheema added 3 commits December 12, 2023 16:14

Merge branch 'main' into saml-cypress-integration

5a6cca0

fix

fa83e7c

adding a comment tracking the new skipped test

618ad00

MadameSheema self-assigned this Dec 12, 2023

MadameSheema added release_note:skip Skip the PR/issue when compiling release notes Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.13.0 labels Dec 12, 2023

MadameSheema marked this pull request as ready for review December 12, 2023 16:05

MadameSheema requested review from patrykkopycinski, oatkiller, maximpn, banderror and a team as code owners December 12, 2023 16:05

patrykkopycinski approved these changes Dec 12, 2023

View reviewed changes

stephmilovic reviewed Dec 12, 2023

View reviewed changes

x-pack/test/security_solution_cypress/cypress/e2e/explore/overview/overview.cy.ts Show resolved Hide resolved

charlie-pichette approved these changes Dec 12, 2023

View reviewed changes

dkirchan approved these changes Dec 12, 2023

View reviewed changes

stephmilovic approved these changes Dec 12, 2023

View reviewed changes

cavokz approved these changes Dec 12, 2023

View reviewed changes

Merge branch 'main' into saml-cypress-integration

4dd7911

delanni reviewed Dec 13, 2023

View reviewed changes

delanni approved these changes Dec 13, 2023

View reviewed changes

vitaliidm approved these changes Dec 13, 2023

View reviewed changes

MadameSheema merged commit c580213 into elastic:main Dec 13, 2023
41 checks passed

kibanamachine added the backport:skip This commit does not require backporting label Dec 13, 2023

MadameSheema deleted the saml-cypress-integration branch December 13, 2023 16:13

MadameSheema mentioned this pull request Dec 13, 2023

[Security Solution] Preparing Cypress for the second quality gate - latest steps #173327

Open

43 tasks

mgiota mentioned this pull request May 17, 2024

[Observability solution] [SLO] Run burn rate api tests in serverless & ess using mocha tagging #183113

Closed

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] [Cypress] SAML for Serverless login and role testing #172655

[Security Solution] [Cypress] SAML for Serverless login and role testing #172655

MadameSheema commented Dec 6, 2023 •

edited

Loading

pheyos left a comment

MadameSheema commented Dec 6, 2023

MadameSheema commented Dec 11, 2023

MadameSheema commented Dec 11, 2023

MadameSheema commented Dec 11, 2023

MadameSheema commented Dec 12, 2023

MadameSheema commented Dec 12, 2023

elasticmachine commented Dec 12, 2023

MadameSheema commented Dec 12, 2023

charlie-pichette left a comment

dkirchan Dec 12, 2023

MadameSheema Dec 12, 2023

stephmilovic left a comment

kibana-ci commented Dec 12, 2023

delanni Dec 13, 2023

MadameSheema Dec 13, 2023

dkirchan Dec 13, 2023

delanni left a comment

vitaliidm left a comment

vitaliidm Dec 13, 2023

MadameSheema Dec 13, 2023

MadameSheema commented Dec 13, 2023

[Security Solution] [Cypress] SAML for Serverless login and role testing #172655

[Security Solution] [Cypress] SAML for Serverless login and role testing #172655

Conversation

MadameSheema commented Dec 6, 2023 • edited Loading

Summary

How to test it in your machine

Serverless FTR

Serverless MKI

pheyos left a comment

Choose a reason for hiding this comment

MadameSheema commented Dec 6, 2023

MadameSheema commented Dec 11, 2023

MadameSheema commented Dec 11, 2023

MadameSheema commented Dec 11, 2023

MadameSheema commented Dec 12, 2023

MadameSheema commented Dec 12, 2023

elasticmachine commented Dec 12, 2023

MadameSheema commented Dec 12, 2023

charlie-pichette left a comment

Choose a reason for hiding this comment

dkirchan Dec 12, 2023

Choose a reason for hiding this comment

MadameSheema Dec 12, 2023

Choose a reason for hiding this comment

stephmilovic left a comment

Choose a reason for hiding this comment

kibana-ci commented Dec 12, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

delanni Dec 13, 2023

Choose a reason for hiding this comment

MadameSheema Dec 13, 2023

Choose a reason for hiding this comment

dkirchan Dec 13, 2023

Choose a reason for hiding this comment

delanni left a comment

Choose a reason for hiding this comment

vitaliidm left a comment

Choose a reason for hiding this comment

vitaliidm Dec 13, 2023

Choose a reason for hiding this comment

MadameSheema Dec 13, 2023

Choose a reason for hiding this comment

MadameSheema commented Dec 13, 2023

MadameSheema commented Dec 6, 2023 •

edited

Loading