[Security Solution][Endpoint] Add support to Action Responder script for `get-file` response action #142663

paul-tavares · 2022-10-04T19:35:45Z

Summary

Adds support to Action Responder script (x-pack/plugins/security_solution/scripts/endpoint/endpoint_action_responder.js) for get-file response action (coming in 8.6). Will write a generated file to ES when action is submitted
Alters the .generateResponse() method of EndpointActionGenerator to also generate parameters and output for get-file action
Adds type for UploadedFile and const's for the indexes that will store the files.

NOTE: There is no way to test this just yet - unless one creates an action manually via ES and then runs the endpoint action responder script. A subsequent PR will introduce the command into the console.

elasticmachine · 2022-10-04T19:35:48Z

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

paul-tavares · 2022-10-04T19:36:37Z

x-pack/plugins/security_solution/common/endpoint/data_generators/endpoint_action_generator.ts

          },
-          // randomly before a few hours/minutes/seconds later
-          started_at: new Date(startedAtTimes[this.randomN(startedAtTimes.length)]).toISOString(),
-          output: undefined,


output seemed to be in the wrong place here. It should have been under data

paul-tavares · 2022-10-04T19:38:16Z

x-pack/plugins/security_solution/common/endpoint/types/file_storage.ts

+ * The Metadata information about a file that was uploaded by Endpoint
+ * as a result of a `get-file` response action
+ */
+export interface UploadedFile {


FYI - having had some more time looking at the file upload and how we will use it in the UI, I'm not sure that we will need this Type. Or we might needed it, but not under common/ - maybe on the server side only. We might not actually need to interact with the file stored because the Endpoint action response will likely have everything we need.

paul-tavares · 2022-10-04T19:38:55Z

x-pack/plugins/security_solution/common/endpoint/types/file_storage.ts

+ * as a result of a `get-file` response action
+ */
+export interface UploadedFile {
+  file: {


I only defined a subset of the ECS file attributes. Let me now if I should define others and/or the full set.

paul-tavares · 2022-10-04T19:39:44Z

x-pack/plugins/security_solution/scripts/endpoint/action_responder/utils.ts

+    // Add the file's metadata
+    const fileMeta = await esClient.index<UploadedFile>({
+      index: FILE_STORAGE_METADATA_INDEX,
+      id: `${action.id}.${action.hosts[0]}`,


@pzl - I think this is right for now, correct?
If anything changes, and you remember, let us know and we'll update.

correct for now. This is a completely opaque value that we get to determine. So whatever order or value is easiest for us to look up again.

either approach works IMO. both pieces of data is on the Action request/response

paul-tavares · 2022-10-04T19:41:22Z

x-pack/plugins/security_solution/scripts/endpoint/action_responder/utils.ts

+
+    await esClient.index({
+      index: FILE_STORAGE_DATA_INDEX,
+      id: `${fileMeta._id}.0`,


@pzl - I think this is right. I did not have a file with multiple chunks, but looking at the data you sent me, I assumed that each chunk's id would be comprised of the File's meta document _id + .<chunk_number>.

precisely correct

paul-tavares · 2022-10-04T19:41:53Z

x-pack/plugins/security_solution/scripts/endpoint/action_responder/utils.ts

+      id: `${fileMeta._id}.0`,
+      body: {
+        bid: fileMeta._id,
+        last: false,


Is this correct for file stored all within one chunk?

no, what I sent you actually had a bug in it. Should still be true for 1-chunk-docs

pzl · 2022-10-04T20:11:00Z

x-pack/plugins/security_solution/common/endpoint/types/file_storage.ts

+      ssdeep?: string;
+      tlsh?: string;


I think it's highly unlikely we will ever use these hashes. After determining the capabilities of agent/endpoint to generate hashes, we will support a specific subset, since fleet server will also need to perform that same hashing, it will be a known subset we support. But is yet to be determined until they start work

md5, sha256 is probably a safe assumption. unless we need to add something simpler like crc

will adjust once we know that.

pzl · 2022-10-04T20:13:14Z

x-pack/plugins/security_solution/common/endpoint/types/file_storage.ts

+     *    of the contents and metadata happens. Deleted files should be treated as if they don’t
+     *    exist. Only files in READY state can transition into DELETED state.
+     */
+    Status: 'AWAITING_UPLOAD' | 'UPLOADING' | 'READY' | 'UPLOAD_ERROR' | 'DELETED';


pzl · 2022-10-04T20:16:45Z

x-pack/plugins/security_solution/scripts/endpoint/action_responder/utils.ts

+      body: {
+        bid: fileMeta._id,
+        last: false,
+        data: 'UEsDBBQACAAIAFVeRFUAAAAAAAAAABMAAAAMACAAYmFkX2ZpbGUudHh0VVQNAAdTVjxjU1Y8Y1NWPGN1eAsAAQT1AQAABBQAAAArycgsVgCiRIWkxBSFtMycVC4AUEsHCKkCwMsTAAAAEwAAAFBLAQIUAxQACAAIAFVeRFWpAsDLEwAAABMAAAAMACAAAAAAAAAAAACkgQAAAABiYWRfZmlsZS50eHRVVA0AB1NWPGNTVjxjU1Y8Y3V4CwABBPUBAAAEFAAAAFBLBQYAAAAAAQABAFoAAABtAAAAAAA=',


for 💩 and 🤣 I actually checked what this is.

It's a zip of a single file, bad_file.txt with contents: this is a bad file.

LOL - yep 😎

(and thanks for testing that out)

kibana-ci · 2022-10-04T23:14:08Z

💚 Build Succeeded

Buildkite Build
Commit: 37d4588

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	6.6MB	6.6MB	+11.0B

History

💔 Build #78119 failed 1ef2b24
💔 Build #78097 failed 954c2d0

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @paul-tavares

ashokaditya

Also please add the get-file command to x-pack/plugins/security_solution/public/management/components/endpoint_response_actions_list/components/use_action_history_url_params.test.ts tests

ashokaditya · 2022-10-05T13:14:31Z

x-pack/plugins/security_solution/common/endpoint/types/file_storage.ts

+      ssdeep?: string;
+      tlsh?: string;
+    };
+    mime_type?: string;


Should we restrict this to the allowed mime types?

yeah, I think we could since the file the endpoint will upload will always be zip. I'll wait on that until we actually get this working with the Endpoint and get agreement on the expectation.

I see. 👍

ashokaditya · 2022-10-05T13:14:56Z

...on/public/management/components/endpoint_response_actions_list/response_actions_log.test.tsx

+      expect(filterList.querySelectorAll('ul>li').length).toEqual(RESPONSE_ACTION_COMMANDS.length);
      expect(
        Array.from(filterList.querySelectorAll('ul>li')).map((option) => option.textContent)
-      ).toEqual(['isolate', 'release', 'kill-process', 'suspend-process', 'processes']);
+      ).toEqual(['isolate', 'release', 'kill-process', 'suspend-process', 'processes', 'get-file']);


paul-tavares · 2022-10-05T13:59:16Z

@ashokaditya

Re: adding to test "...components/endpoint_response_actions_list/components/use_action_history_url_params.test.ts"

I will do this in the next PR. I would like to do some refactoring and move some of the const associated with Response actions to common/service/constants.ts so that they are all in one place. I would like to rename them too in order to clarify what they are.

ashokaditya

🚀 🚢

nicpenning · 2022-10-08T19:08:33Z

Awesome, I am excited for this feature. Since I can't quite follow the complexities of the code, I would like to ask:

Does the end goal .zip file contain a password and if so is it configurable?

…for `get-file` response action (elastic#142663) * Add `const`'s for file storage indexes * Add file storage TS type * Add support to Action Responder script for `get-file` response action * Add support to action generator to output `get-file` responses * correct `last` property of the upload chunk * Fix type * fix UI jest test failures * Fix server jest tests and Generator bug

paul-tavares · 2022-10-11T13:09:33Z

@nicpenning - yes, the zip file that is uploaded by the Endpoint with the requested file will have a password set on it and currently we are not planning on making that configurable. We are planning on using elastic as the password.

nicpenning · 2022-10-11T13:49:19Z

Awesome, thanks! That should be great.

…for `get-file` response action (elastic#142663) * Add `const`'s for file storage indexes * Add file storage TS type * Add support to Action Responder script for `get-file` response action * Add support to action generator to output `get-file` responses * correct `last` property of the upload chunk * Fix type * fix UI jest test failures * Fix server jest tests and Generator bug

paul-tavares added 4 commits October 4, 2022 11:11

Add const's for file storage indexes

ad4caee

Add file storage TS type

bce3567

Add support to Action Responder script for get-file response action

77545dc

Add support to action generator to output get-file responses

954c2d0

paul-tavares added release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v8.6.0 labels Oct 4, 2022

paul-tavares requested a review from a team as a code owner October 4, 2022 19:35

paul-tavares self-assigned this Oct 4, 2022

paul-tavares requested a review from a team as a code owner October 4, 2022 19:35

paul-tavares requested review from ashokaditya and joeypoon October 4, 2022 19:35

paul-tavares requested a review from pzl October 4, 2022 19:35

paul-tavares commented Oct 4, 2022

View reviewed changes

pzl reviewed Oct 4, 2022

View reviewed changes

paul-tavares added 2 commits October 4, 2022 16:19

correct last property of the upload chunk

8f68a8a

Fix type

1ef2b24

paul-tavares requested a review from pzl October 4, 2022 20:29

pzl approved these changes Oct 4, 2022

View reviewed changes

paul-tavares added 2 commits October 4, 2022 18:00

fix UI jest test failures

2d23064

Fix server jest tests and Generator bug

37d4588

kevinlog approved these changes Oct 5, 2022

View reviewed changes

ashokaditya reviewed Oct 5, 2022

View reviewed changes

paul-tavares requested a review from ashokaditya October 5, 2022 13:59

ashokaditya approved these changes Oct 5, 2022

View reviewed changes

paul-tavares merged commit 3fa8a87 into elastic:main Oct 5, 2022

paul-tavares deleted the task/olm-add-generator-get-file-support branch October 5, 2022 14:01

kibanamachine added the backport:skip This commit does not require backporting label Oct 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Endpoint] Add support to Action Responder script for `get-file` response action #142663

[Security Solution][Endpoint] Add support to Action Responder script for `get-file` response action #142663

paul-tavares commented Oct 4, 2022

elasticmachine commented Oct 4, 2022

paul-tavares Oct 4, 2022

paul-tavares Oct 4, 2022

paul-tavares Oct 4, 2022

paul-tavares Oct 4, 2022

pzl Oct 4, 2022

paul-tavares Oct 4, 2022

paul-tavares Oct 4, 2022

pzl Oct 4, 2022

paul-tavares Oct 4, 2022

pzl Oct 4, 2022

pzl Oct 4, 2022

paul-tavares Oct 4, 2022

pzl Oct 4, 2022

pzl Oct 4, 2022

paul-tavares Oct 4, 2022

kibana-ci commented Oct 4, 2022

ashokaditya left a comment

ashokaditya Oct 5, 2022

paul-tavares Oct 5, 2022

ashokaditya Oct 5, 2022

ashokaditya Oct 5, 2022

paul-tavares commented Oct 5, 2022

ashokaditya left a comment

nicpenning commented Oct 8, 2022

paul-tavares commented Oct 11, 2022

nicpenning commented Oct 11, 2022

[Security Solution][Endpoint] Add support to Action Responder script for get-file response action #142663

[Security Solution][Endpoint] Add support to Action Responder script for get-file response action #142663

Conversation

paul-tavares commented Oct 4, 2022

Summary

elasticmachine commented Oct 4, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibana-ci commented Oct 4, 2022

💚 Build Succeeded

Metrics [docs]

Async chunks

History

ashokaditya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paul-tavares commented Oct 5, 2022

ashokaditya left a comment

Choose a reason for hiding this comment

nicpenning commented Oct 8, 2022

paul-tavares commented Oct 11, 2022

nicpenning commented Oct 11, 2022

[Security Solution][Endpoint] Add support to Action Responder script for `get-file` response action #142663

[Security Solution][Endpoint] Add support to Action Responder script for `get-file` response action #142663