Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Defend Workflows][E2E]Get file command from response console #156159

Merged
merged 73 commits into from
May 16, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
2f884d5
avatar aria label
szwarckonrad Apr 4, 2023
704886c
isolate command e2e coverage
szwarckonrad Apr 5, 2023
feab480
Merge branch 'main' into endpoint-isolate-e2e-coverage
szwarckonrad Apr 5, 2023
5db6e07
typings
szwarckonrad Apr 6, 2023
51b63c3
Merge branch 'main' into endpoint-isolate-e2e-coverage
szwarckonrad Apr 6, 2023
4de9906
typings
szwarckonrad Apr 6, 2023
67eb92c
cleanup
szwarckonrad Apr 17, 2023
12874a1
Merge branch 'main' into endpoint-isolate-e2e-coverage
szwarckonrad Apr 17, 2023
fcc702f
use custom document generator
szwarckonrad Apr 17, 2023
25674ff
Merge branch 'main' into endpoint-isolate-e2e-coverage
szwarckonrad Apr 17, 2023
7c9043f
Merge branch 'main' into endpoint-isolate-e2e-coverage
szwarckonrad Apr 18, 2023
c54509b
manualy refresh result list
szwarckonrad Apr 18, 2023
d2fbb5d
Merge branch 'main' into endpoint-isolate-e2e-coverage
szwarckonrad Apr 18, 2023
9415935
remove artifacts after endpoints.cy.ts test
szwarckonrad Apr 18, 2023
ac9e110
Merge branch 'main' into endpoint-isolate-e2e-coverage
szwarckonrad Apr 18, 2023
193968b
backport isolate e2e tests to multipass
szwarckonrad Apr 20, 2023
f84233b
Merge branch 'main' into endpoint-isolate-e2e-coverage
szwarckonrad Apr 20, 2023
b3a7460
cleanup
szwarckonrad Apr 20, 2023
345c733
Merge branch 'endpoint-isolate-e2e-coverage' into endpoint-isolate-e2…
szwarckonrad Apr 20, 2023
b5b6941
tweaks
szwarckonrad Apr 21, 2023
62fa777
Merge branch 'main' into endpoint-isolate-e2e-coverage
szwarckonrad Apr 21, 2023
5157dda
Merge branch 'endpoint-isolate-e2e-coverage' into endpoint-isolate-e2…
szwarckonrad Apr 21, 2023
7c13a7e
cleanup
szwarckonrad Apr 21, 2023
890c52c
test isolate and processes commands
szwarckonrad Apr 21, 2023
678aaa7
Merge branch 'main' into endpoint-e2e-coverage-multipass
szwarckonrad Apr 24, 2023
7aea2b5
Merge branch 'main' into endpoint-isolate-e2e-coverage-multipass
szwarckonrad Apr 24, 2023
2b07574
type returns of helper functions
szwarckonrad Apr 24, 2023
7b8fe2a
Merge branch 'endpoint-isolate-e2e-coverage-multipass' into endpoint-…
szwarckonrad Apr 24, 2023
12b16d2
tweaks
szwarckonrad Apr 24, 2023
aef3b55
Merge branch 'main' into endpoint-e2e-coverage-multipass
szwarckonrad Apr 24, 2023
c6c32bd
Merge branch 'main' into endpoint-e2e-response-console
szwarckonrad Apr 25, 2023
68f07c9
test
szwarckonrad Apr 25, 2023
7d03f5c
Merge branch 'main' into endpoint-e2e-coverage-multipass
szwarckonrad Apr 25, 2023
d588d6d
fix action
patrykkopycinski Apr 25, 2023
f5ce24d
divide endpoint list checking function
szwarckonrad Apr 25, 2023
9a6c19d
Merge branch 'endpoint-e2e-coverage-multipass' into endpoint-e2e-resp…
szwarckonrad Apr 25, 2023
71a138e
e2e coverage
szwarckonrad Apr 25, 2023
44e0b5c
Merge branch 'main' into endpoint-e2e-response-console
szwarckonrad Apr 25, 2023
28e459e
Merge branch 'main' into endpoint-e2e-coverage-multipass
szwarckonrad Apr 25, 2023
02d2c1c
Merge branch 'main' into endpoint-e2e-response-console
szwarckonrad Apr 25, 2023
2a2ed44
Merge branch 'main' into endpoint-e2e-coverage-multipass
szwarckonrad Apr 25, 2023
983e5dc
Merge branch 'main' into endpoint-e2e-response-console
szwarckonrad Apr 26, 2023
9f74e8f
Merge branch 'main' into endpoint-e2e-coverage-multipass
szwarckonrad Apr 26, 2023
dc61a76
cleanup
szwarckonrad Apr 26, 2023
c0f3f24
naming
szwarckonrad Apr 26, 2023
7f0526f
naming
szwarckonrad Apr 26, 2023
d826943
explicit types
szwarckonrad Apr 26, 2023
bdce83f
Merge branch 'main' into endpoint-e2e-coverage-multipass
szwarckonrad Apr 26, 2023
513c616
Merge branch 'endpoint-e2e-coverage-multipass' into endpoint-e2e-resp…
szwarckonrad Apr 26, 2023
9fef80e
Merge remote-tracking branch 'origin/endpoint-e2e-response-console' i…
szwarckonrad Apr 26, 2023
4090b64
Merge branch 'main' into endpoint-e2e-response-console
szwarckonrad Apr 27, 2023
6e7e9b5
Merge branch 'main' into endpoint-e2e-response-console
szwarckonrad Apr 27, 2023
15c4521
explicit types
szwarckonrad Apr 27, 2023
6b4639f
check for spawned endpoint on the endpoint list as there might be dan…
szwarckonrad Apr 27, 2023
4f8b8a1
move response actions out of emulator scope
szwarckonrad Apr 28, 2023
bedfb41
Merge branch 'main' into endpoint-e2e-response-console
szwarckonrad Apr 28, 2023
40803c5
get file mocked data e2e
szwarckonrad Apr 28, 2023
0809f48
Merge branch 'main' into endpoint-e2e-get-file
szwarckonrad Apr 28, 2023
1aec910
Merge branch 'main' into endpoint-e2e-get-file
szwarckonrad Apr 28, 2023
cf905bc
Merge branch 'main' into endpoint-e2e-get-file
szwarckonrad Apr 30, 2023
9eb8a10
Merge branch 'main' into endpoint-e2e-get-file
szwarckonrad May 8, 2023
5da33e6
get file real endpoint e2e
szwarckonrad May 8, 2023
dd9e155
Merge remote-tracking branch 'origin/endpoint-e2e-get-file' into endp…
szwarckonrad May 8, 2023
cb4aa5f
Merge branch 'main' into endpoint-e2e-get-file
szwarckonrad May 8, 2023
b78b78a
tweaks
szwarckonrad May 8, 2023
e69493e
Merge remote-tracking branch 'origin/endpoint-e2e-get-file' into endp…
szwarckonrad May 8, 2023
f74767f
unskip response actions cy
szwarckonrad May 11, 2023
c6bf892
Merge remote-tracking branch 'upstream/main'
szwarckonrad May 15, 2023
61a8a8f
Merge branch 'main' into endpoint-e2e-get-file
szwarckonrad May 15, 2023
76ff802
Merge branch 'main' into endpoint-e2e-get-file
szwarckonrad May 16, 2023
fe4e325
Merge branch 'main' into endpoint-e2e-get-file
szwarckonrad May 16, 2023
2a1a705
Merge branch 'main' into endpoint-e2e-get-file
szwarckonrad May 16, 2023
058cb51
Merge branch 'main' into endpoint-e2e-get-file
szwarckonrad May 16, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,30 @@ declare global {
arg: { endpointAgentIds: string[] },
options?: Partial<Loggable & Timeoutable>
): Chainable<DeleteAllEndpointDataResponse>;

task(
name: 'createFileOnEndpoint',
arg: { hostname: string; path: string; content: string },
options?: Partial<Loggable & Timeoutable>
): Chainable<null>;

task(
name: 'uploadFileToEndpoint',
arg: { hostname: string; srcPath: string; destPath: string },
options?: Partial<Loggable & Timeoutable>
): Chainable<null>;

task(
name: 'installPackagesOnEndpoint',
arg: { hostname: string; packages: string[] },
options?: Partial<Loggable & Timeoutable>
): Chainable<null>;

task(
name: 'readZippedFileContentOnEndpoint',
arg: { hostname: string; path: string; password?: string },
options?: Partial<Loggable & Timeoutable>
): Chainable<string>;
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,74 @@ describe('Response console', () => {
});
});

describe('User journey for Get file command', () => {
let response: IndexedFleetEndpointPolicyResponse;
let initialAgentData: Agent;

const fileContent = 'This is a test file for the get-file command.';
const filePath = `/home/ubuntu/test_file.txt`;

before(() => {
getAgentByHostName(endpointHostname).then((agentData) => {
initialAgentData = agentData;
});

getEndpointIntegrationVersion().then((version) =>
createAgentPolicyTask(version).then((data) => {
response = data;
})
);

cy.task('installPackagesOnEndpoint', { hostname: endpointHostname, packages: ['unzip'] });

cy.task('createFileOnEndpoint', {
hostname: endpointHostname,
path: filePath,
content: fileContent,
});
});
Comment on lines +189 to +196
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This really feels overkill... All you are trying to test is that the endpoint can return a file based on a get-file action. Why not just retrieve a log file from the directory where endpoint is installed?

I'm ok if you leave it, but it just feels like we're doing unnecessary work IMO

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I was aiming at is an e2e coverage of user's journey, therefore I want to make sure user can download and extract the exact file they requested. Do you thinks that's going too far? If so then the rest of the changes doesn't make much sense indeed.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you can make sure that the user can retrieve a file from the host - goal here should be to ensure Endpoint is sending back the file to ensure we have round trip communication. to do that, you can just get-file and use the path to any file on the VM - example: one of the log files under the Agent's install directory.

I was just really confused as to why you were creatingFileOnEndpoint() and uploadFileToEndpoint() and readZipFileContentOnEndpoint(). Just feels all of these are not necessary

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I went this way because I wanted to confirm that user is getting the file they requested and that the archive can be extracted with default password and I wouIdn't be able to do that with a random log file. In the process I found out that unzipping password protected zips in node can be tricky, its not platform agnostic when done locally so the most "stable" way is to do it on the host that runs ubuntu. It is cumbersome for sure but in the end its cheap time wise and covers what I believe is worth checking. It all comes down to the question whether checking the archive content adds value or not. Let me know what you think about it :)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its feels like you are testing the Endpoint and not really focusing on the test from Kibana. IMO, this is unnecessary and goes beyond what we should be focusing on. Go ahead an keep it if you feel strongly about this, but remember that we are not testing endpoint - we're testing that we can communicate with the endpoint for different actions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@paul-tavares , isn't e2e test's purpose to validate if the user journey is successful? We don't specifically test if just one part of the functionality works.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have different understanding of the level of testing we are validating with these real endpoint e2e tests. You seem to feel strongly that this is the right level so I'm going to retreat 😄 and let others provide feedback.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since e2e tests are slow to write and run I would argue that we write tests that cover only the essential high-value flow for a user. In other words, we should only test high-value paths in the e2e tests that we can't test in unit/integration tests.

So in this example, it is enough to test that I can fetch an actual file from the endpoint and that I have a downloadable link, any file will do. I am not interested in verifying what the user does with it after it is fetched on the console. In terms of other console actions as well, it is enough to write a test that verifies that the action does what it is supposed to do. Send a request and check the response from the endpoint is what is expected.

I do understand the appeal of writing a bunch of tests to verify the entire click-by-click user interaction, but that would mean duplicating a bunch of tests that already cover that. Considering the effort required to write, maintain, and debug tests, I would argue against adding redundant tests for the whole user interaction.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey all, thanks for the discussion, great to see us thinking deeply about what and how we test.

Here's my thoughts:

  • The main goals of testing with real Agent/Endpoints is to validate that we are integrating correctly with Endpoints and that the data flow is working end to end. In this case, we want to make sure that we can successfully send the Get File command to the Endpoint and it will respond back successfully. That being said, I think it is sufficient to just look at data flow for many of these tests and we don't necessarily need to validate user action on the host itself.
  • However, on the other hand, I certainly see the value to take the user journey flow one step further and validate that downloads, etc are working correctly. Thinking about it in terms of a user journey as @tomsonpl and @szwarckonrad laid out.

Considering the above, we definitely achieve the initial goal of testing stack/Endpoint data flow ✅

I think validating that the user can download the file from Kibana is also something good to do here. Endpoint tests would not cover this particular scenario in the user journey as they are focused on Endpoint functionality itself and not what happens after file upload to ES is complete. So I think the extra coverage would be good here ✅

I think we should merge this test as is because it does automate a scenario that we would otherwise do manually. If I were to test this manually, I would certainly take the extra step to download and inspect the file. As we strive to eliminate manual testing, this seems like something good to do.

I acknowledge that this goes beyond our original thoughts around testing this area, but I see the value in these extra steps. I think it's good to always evolve the way we think about testing, especially early on with a new framework.


after(() => {
if (initialAgentData?.policy_id) {
reassignAgentPolicy(initialAgentData.id, initialAgentData.policy_id);
}
if (response) {
cy.task('deleteIndexedFleetEndpointPolicies', response);
}
});

it('"get-file --path" - should retrieve a file', () => {
waitForEndpointListPageToBeLoaded(endpointHostname);
openResponseConsoleFromEndpointList();
inputConsoleCommand(`get-file --path ${filePath}`);
submitCommand();
cy.getByTestSubj('getFileSuccess', { timeout: 60000 }).within(() => {
cy.contains('File retrieved from the host.');
cy.contains('(ZIP file passcode: elastic)');
cy.contains(
'Files are periodically deleted to clear storage space. Download and save file locally if needed.'
);
cy.contains('Click here to download').click();
const downloadsFolder = Cypress.config('downloadsFolder');
cy.readFile(`${downloadsFolder}/upload.zip`);

cy.task('uploadFileToEndpoint', {
hostname: endpointHostname,
srcPath: `${downloadsFolder}/upload.zip`,
destPath: '/home/ubuntu/upload.zip',
});

cy.task('readZippedFileContentOnEndpoint', {
hostname: endpointHostname,
path: '/home/ubuntu/upload.zip',
password: 'elastic',
}).then((unzippedFileContent) => {
expect(unzippedFileContent).to.equal(fileContent);
});
paul-tavares marked this conversation as resolved.
Show resolved Hide resolved
});
});
});

describe('document signing', () => {
let response: IndexedFleetEndpointPolicyResponse;
let initialAgentData: Agent;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -227,4 +227,53 @@ describe('Response console', () => {
cy.contains('Action completed.', { timeout: 120000 }).should('exist');
});
});

describe('Get file command', () => {
let endpointData: ReturnTypeFromChainable<typeof indexEndpointHosts>;
let endpointHostname: string;
let getFileRequestResponse: ActionDetails;

before(() => {
indexEndpointHosts({ withResponseActions: false, isolation: false }).then(
(indexEndpoints) => {
endpointData = indexEndpoints;
endpointHostname = endpointData.data.hosts[0].host.name;
}
);
});

after(() => {
if (endpointData) {
endpointData.cleanup();
// @ts-expect-error ignore setting to undefined
endpointData = undefined;
}
});

it('should get file from response console', () => {
waitForEndpointListPageToBeLoaded(endpointHostname);
openResponseConsoleFromEndpointList();
inputConsoleCommand(`get-file --path /test/path/test.txt`);

interceptActionRequests((responseBody) => {
getFileRequestResponse = responseBody;
}, 'get-file');
submitCommand();
cy.contains('Retrieving the file from host.').should('exist');
cy.wait('@get-file').then(() => {
sendActionResponse(getFileRequestResponse);
});
cy.getByTestSubj('getFileSuccess').within(() => {
cy.contains('File retrieved from the host.');
cy.contains('(ZIP file passcode: elastic)');
cy.contains(
'Files are periodically deleted to clear storage space. Download and save file locally if needed.'
);
cy.contains('Click here to download').click();
});

const downloadsFolder = Cypress.config('downloadsFolder');
cy.readFile(`${downloadsFolder}/upload.zip`);
});
});
paul-tavares marked this conversation as resolved.
Show resolved Hide resolved
});
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@
// / <reference types="cypress" />

import type { CasePostRequest } from '@kbn/cases-plugin/common/api';
import execa from 'execa';
import {
sendEndpointActionResponse,
sendFleetActionResponse,
Expand Down Expand Up @@ -224,6 +225,70 @@ export const dataLoadersForRealEndpoints = (
return destroyEndpointHost(kbnClient, createdHost).then(() => null);
},

createFileOnEndpoint: async ({
paul-tavares marked this conversation as resolved.
Show resolved Hide resolved
hostname,
path,
content,
}: {
hostname: string;
path: string;
content: string;
}): Promise<null> => {
await execa(`multipass`, ['exec', hostname, '--', 'sh', '-c', `echo ${content} > ${path}`]);
return null;
},

uploadFileToEndpoint: async ({
hostname,
srcPath,
destPath = '.',
}: {
hostname: string;
srcPath: string;
destPath: string;
}): Promise<null> => {
await execa(`multipass`, ['transfer', srcPath, `${hostname}:${destPath}`]);
return null;
},

installPackagesOnEndpoint: async ({
hostname,
packages,
}: {
hostname: string;
packages: string[];
}): Promise<null> => {
await execa(`multipass`, [
'exec',
hostname,
'--',
'sh',
'-c',
`sudo apt install -y ${packages.join(' ')}`,
]);
return null;
},

readZippedFileContentOnEndpoint: async ({
hostname,
path,
password,
}: {
hostname: string;
path: string;
password?: string;
}): Promise<string> => {
const result = await execa(`multipass`, [
'exec',
hostname,
'--',
'sh',
'-c',
`unzip -p ${password ? `-P ${password} ` : ''}${path}`,
]);
return result.stdout;
},
paul-tavares marked this conversation as resolved.
Show resolved Hide resolved

stopEndpointHost: async () => {
const hosts = await getEndpointHosts();
const hostName = hosts[0].name;
Expand Down