[Defend Workflows][E2E]Get file command from response console

[szwarckonrad]

This PR tackles get-file command issued from response console. Two tests are included:

1. x-pack/plugins/security_solution/public/management/cypress/e2e/mocked_data/response_console.cy.ts mocked data
2. x-pack/plugins/security_solution/public/management/cypress/e2e/endpoint/response_console.cy.ts real endpoint

[elasticmachine]

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

[paul-tavares]

This really feels overkill... All you are trying to test is that the endpoint can return a file based on a get-file action. Why not just retrieve a log file from the directory where endpoint is installed?

I'm ok if you leave it, but it just feels like we're doing unnecessary work IMO

[szwarckonrad]

What I was aiming at is an e2e coverage of user's journey, therefore I want to make sure user can download and extract the exact file they requested. Do you thinks that's going too far? If so then the rest of the changes doesn't make much sense indeed.

[paul-tavares]

Yes, you can make sure that the user can retrieve a file from the host - goal here should be to ensure Endpoint is sending back the file to ensure we have round trip communication. to do that, you can just get-file and use the path to any file on the VM - example: one of the log files under the Agent's install directory.

I was just really confused as to why you were creatingFileOnEndpoint() and uploadFileToEndpoint() and readZipFileContentOnEndpoint(). Just feels all of these are not necessary

[szwarckonrad]

Yes, I went this way because I wanted to confirm that user is getting the file they requested and that the archive can be extracted with default password and I wouIdn't be able to do that with a random log file. In the process I found out that unzipping password protected zips in node can be tricky, its not platform agnostic when done locally so the most "stable" way is to do it on the host that runs ubuntu. It is cumbersome for sure but in the end its cheap time wise and covers what I believe is worth checking. It all comes down to the question whether checking the archive content adds value or not. Let me know what you think about it :)

[paul-tavares]

Its feels like you are testing the Endpoint and not really focusing on the test from Kibana. IMO, this is unnecessary and goes beyond what we should be focusing on. Go ahead an keep it if you feel strongly about this, but remember that we are not testing endpoint - we're testing that we can communicate with the endpoint for different actions.

[tomsonpl]

@paul-tavares , isn't e2e test's purpose to validate if the user journey is successful? We don't specifically test if just one part of the functionality works.

[paul-tavares]

I think we have different understanding of the level of testing we are validating with these real endpoint e2e tests. You seem to feel strongly that this is the right level so I'm going to retreat 😄 and let others provide feedback.

[ashokaditya]

Since e2e tests are slow to write and run I would argue that we write tests that cover only the essential high-value flow for a user. In other words, we should only test high-value paths in the e2e tests that we can't test in unit/integration tests.

So in this example, it is enough to test that I can fetch an actual file from the endpoint and that I have a downloadable link, any file will do. I am not interested in verifying what the user does with it after it is fetched on the console. In terms of other console actions as well, it is enough to write a test that verifies that the action does what it is supposed to do. Send a request and check the response from the endpoint is what is expected.

I do understand the appeal of writing a bunch of tests to verify the entire click-by-click user interaction, but that would mean duplicating a bunch of tests that already cover that. Considering the effort required to write, maintain, and debug tests, I would argue against adding redundant tests for the whole user interaction.

[kevinlog]

hey all, thanks for the discussion, great to see us thinking deeply about what and how we test.

Here's my thoughts:

• The main goals of testing with real Agent/Endpoints is to validate that we are integrating correctly with Endpoints and that the data flow is working end to end. In this case, we want to make sure that we can successfully send the Get File command to the Endpoint and it will respond back successfully. That being said, I think it is sufficient to just look at data flow for many of these tests and we don't necessarily need to validate user action on the host itself.
• However, on the other hand, I certainly see the value to take the user journey flow one step further and validate that downloads, etc are working correctly. Thinking about it in terms of a user journey as @tomsonpl and @szwarckonrad laid out.

Considering the above, we definitely achieve the initial goal of testing stack/Endpoint data flow ✅

I think validating that the user can download the file from Kibana is also something good to do here. Endpoint tests would not cover this particular scenario in the user journey as they are focused on Endpoint functionality itself and not what happens after file upload to ES is complete. So I think the extra coverage would be good here ✅

I think we should merge this test as is because it does automate a scenario that we would otherwise do manually. If I were to test this manually, I would certainly take the extra step to download and inspect the file. As we strive to eliminate manual testing, this seems like something good to do.

I acknowledge that this goes beyond our original thoughts around testing this area, but I see the value in these extra steps. I think it's good to always evolve the way we think about testing, especially early on with a new framework.

[tomsonpl]

LGTM

[kibana-ci]

💛 Build succeeded, but was flaky

• Buildkite Build
• Commit: 058cb51

Failed CI Steps

• Defend Workflows Cypress Tests

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
enterpriseSearch	19	21	+2
securitySolution	400	404	+4
total			+6

Total ESLint disabled count

id	before	after	diff
enterpriseSearch	20	22	+2
securitySolution	480	484	+4
total			+6

History

• 💔 Build #128230 failed 2a1a705
• 💛 Build #128098 was flaky fe4e325
• 💚 Build #128081 succeeded 76ff802
• 💔 Build #127895 failed 61a8a8f
• 💛 Build #126189 was flaky e69493e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[kibanamachine]

💔 All backports failed

Status	Branch	Result
❌	8.8	Backport failed because of merge conflicts You might need to backport the following PRs to 8.8: - [Defend Workflows][E2E]Endpoint e2e response console (#155605)

Manual backport

To create the backport manually run:

node scripts/backport --pr 156159

Questions ?

Please refer to the Backport tool documentation

[szwarckonrad]

💚 All backports created successfully

Status	Branch	Result
✅	8.8

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation