[Security solution] [Endpoint] Fix bad artifact migration #111294

dasansol92 · 2021-09-06T15:16:44Z

Summary

Zip compression was removed during artifacts generation as it was done on the fleet side.
We need to keep this behaviour for the migration script as it is taking the old ones already zip compressed.

fixes #111210

For maintainers

This was checked for breaking API changes and was labeled appropriately

…O artifact

…e creating the new one from fleet

…tArtifact and deleteArtifact as there is an ES error in those methods if not mocked

dasansol92 · 2021-09-07T09:45:34Z

@elasticmachine merge upstream

elasticmachine · 2021-09-07T12:26:43Z

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

paul-tavares · 2021-09-07T12:31:51Z

x-pack/plugins/fleet/server/services/artifacts/mocks.ts

@@ -77,10 +82,12 @@ export const generateEsRequestErrorApiResponseMock = (
  );
 };

-export const generateArtifactEsGetSingleHitMock = (): SearchHit<ArtifactElasticsearchProperties> => {
+export const generateArtifactEsGetSingleHitMock = (


FYI: Just making this mock provider a bit more robust by accepting on input an optional artifact

elasticmachine · 2021-09-07T12:31:55Z

Pinging @elastic/fleet (Team:Fleet)

paul-tavares

🔥 🔥

academo · 2021-09-07T12:38:33Z

x-pack/plugins/security_solution/server/endpoint/lib/artifacts/migrate_artifacts_to_fleet.ts

+          artifact.attributes.body = (
+            await inflateAsync(Buffer.from(artifact.attributes.body, 'base64'))
+          ).toString('base64');
+        }


is there a case where is not compressed? if so, do we have tests to cover it?

@paul-tavares correct me if I'm wrong, but if this script is executed just one time it shouldn't be non compressed artifacts right?

@academo no, there is no case where an Artifact would be stored in the Saved Object with out being compressed. I think this check is just to be safe, but agree is not technically needed.

joshdover

Changes to Fleet mock LGTM

paul-tavares · 2021-09-07T14:13:39Z

...k/plugins/security_solution/server/endpoint/lib/artifacts/migrate_artifacts_to_fleet.test.ts

+      expect.objectContaining({
+        ...soArtifactEntry,
+        compressionAlgorithm: 'zlib',
+        body: 'eJyrVkrNKynKTC1WsoqOrQUAJxkFKQ==',


Just noticed this (thanks for pointing it out.
Can you remove this and instead changing the migration change to not mutate the body prop? like:

if (isCompressed(artifact.attributes)) { artifact.attributes = { ...artifact.attributes, body: ( await inflateAsync(Buffer.from(artifact.attributes.body, 'base64')) ).toString('base64'), }; }

Or - capture a copy of the soArtifatEntry here in the test prior to calling migration (above like 115) and then use that to check again on like 119 (and remove body from here)

paul-tavares · 2021-09-07T14:17:01Z

x-pack/plugins/security_solution/server/endpoint/lib/artifacts/migrate_artifacts_to_fleet.ts

@@ -57,6 +65,15 @@ export const migrateArtifactsToFleet = async (
      }

      for (const artifact of artifactList) {
+        if (isCompressed(artifact.attributes)) {
+          artifact.attributes = {
+            ...artifact.attributes,


paul-tavares

🚢

kibanamachine · 2021-09-07T16:35:18Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💛 Build #151426 was flaky 36fa4cd
💔 Build #151405 failed fe33bb3
💔 Build #151324 failed 77358c7

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @paul-tavares

…1294) * Working test that validate migrated artifact has same properties as SO artifact * Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet Co-authored-by: Paul Tavares <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

kibanamachine · 2021-09-07T16:39:40Z

💚 Backport successful

Status	Branch	Result
✅	7.15
✅	7.x
✅	7.14

The backport PRs will be merged automatically after passing CI.

…111432) * Working test that validate migrated artifact has same properties as SO artifact * Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet Co-authored-by: Paul Tavares <[email protected]> Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: David Sánchez <[email protected]> Co-authored-by: Paul Tavares <[email protected]>

…111430) * Working test that validate migrated artifact has same properties as SO artifact * Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet Co-authored-by: Paul Tavares <[email protected]> Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: David Sánchez <[email protected]> Co-authored-by: Paul Tavares <[email protected]>

…111431) * Working test that validate migrated artifact has same properties as SO artifact * Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet Co-authored-by: Paul Tavares <[email protected]> Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: David Sánchez <[email protected]> Co-authored-by: Paul Tavares <[email protected]>

…-link-to-kibana-app * 'master' of github.com:elastic/kibana: (61 commits) [Logs UI] Fix alert previews for thresholds of `0` (elastic#111150) [Archive Migration][Partial] discover apps-discover (elastic#110437) [APM] Set start date of APM ML job to -4 weeks (elastic#111375) [ML] APM Latency Correlations: Code consolidation. (elastic#110790) [Discover] Fix indices permission for multiline test (elastic#111284) [Detection Rules] Add 7.15 rules (elastic#111464) [Security Solution][Endpoint][Host Isolation] Hide isolate host option in alert details rather than disabling (elastic#111064) React version of angular license view (elastic#111317) [APM] Fix link in readme (elastic#111362) [Security Solution] add agent field to generator (elastic#111428) [Dashboard] Retain Tags on Quicksave (elastic#111015) Reorder App Search ingestion methods (elastic#111361) Port performance docs to new docs system. (elastic#111063) [Security Solution][RAC] Fixes updatedAt loading bug (elastic#111010) [sample data] update web log geo.src field to match country code of geo.coordinates (elastic#110885) [Security solution] [Endpoint] Fix bad artifact migration (elastic#111294) Fix copy typo. (elastic#111203) [build] Remove empty optimize directory (elastic#111393) [Maps] fix term join not updating when editing right field (elastic#111030) [Fleet] Set default settings in component template instead of the index template (elastic#111197) ... # Conflicts: # x-pack/plugins/reporting/public/management/__snapshots__/report_listing.test.tsx.snap # x-pack/plugins/reporting/public/management/report_listing.test.tsx

…1294) * Working test that validate migrated artifact has same properties as SO artifact * Checks if artifact is compressed and uncompress it if necessary before creating the new one from fleet Co-authored-by: Paul Tavares <[email protected]> Co-authored-by: Kibana Machine <[email protected]>

paul-tavares and others added 4 commits September 3, 2021 17:56

attempt to get test done for bad migration (still not working)

8bfe120

Working test that validate migrated artifact has same properties as S…

c92b8ff

…O artifact

Checks if artifact is compressed and uncompress it if necessary befor…

77358c7

…e creating the new one from fleet

Use non mocked instance for createArtifact and mocked instance for ge…

fe33bb3

…tArtifact and deleteArtifact as there is an ES error in those methods if not mocked

Merge branch 'master' into fix/olm-kibana-111210-bad-artifact-migration

36fa4cd

dasansol92 marked this pull request as ready for review September 7, 2021 12:26

dasansol92 requested review from a team as code owners September 7, 2021 12:26

dasansol92 added auto-backport Deprecated - use backport:version if exact versions are needed release_note:fix Team:Defend Workflows “EDR Workflows” sub-team of Security Solution v8.0.0 labels Sep 7, 2021

dasansol92 requested review from a team, academo and paul-tavares and removed request for a team September 7, 2021 12:27

paul-tavares added the v7.16.0 label Sep 7, 2021

paul-tavares reviewed Sep 7, 2021

View reviewed changes

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Sep 7, 2021

paul-tavares approved these changes Sep 7, 2021

View reviewed changes

academo reviewed Sep 7, 2021

View reviewed changes

dasansol92 added v7.14.2 v7.15.0 labels Sep 7, 2021

joshdover approved these changes Sep 7, 2021

View reviewed changes

paul-tavares reviewed Sep 7, 2021

View reviewed changes

Don't mutate data in script

6e13304

paul-tavares reviewed Sep 7, 2021

View reviewed changes

paul-tavares approved these changes Sep 7, 2021

View reviewed changes

academo approved these changes Sep 7, 2021

View reviewed changes

paul-tavares self-assigned this Sep 7, 2021

paul-tavares merged commit 855a338 into elastic:master Sep 7, 2021

kibanamachine mentioned this pull request Sep 7, 2021

[7.15] [Security solution] [Endpoint] Fix bad artifact migration (#111294) #111430

Merged

kibanamachine mentioned this pull request Sep 7, 2021

[7.x] [Security solution] [Endpoint] Fix bad artifact migration (#111294) #111431

Merged

kibanamachine mentioned this pull request Sep 7, 2021

[7.14] [Security solution] [Endpoint] Fix bad artifact migration (#111294) #111432

Merged

kevinlog mentioned this pull request Sep 9, 2021

[Agents]: Artifacts related and "Initial configuration application failed" errors on adding Endpoint Security to the Agent policy. elastic/beats#27064

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security solution] [Endpoint] Fix bad artifact migration #111294

[Security solution] [Endpoint] Fix bad artifact migration #111294

dasansol92 commented Sep 6, 2021 •

edited

Loading

dasansol92 commented Sep 7, 2021

elasticmachine commented Sep 7, 2021

paul-tavares Sep 7, 2021

elasticmachine commented Sep 7, 2021

paul-tavares left a comment

academo Sep 7, 2021

dasansol92 Sep 7, 2021

paul-tavares Sep 7, 2021

joshdover left a comment

paul-tavares Sep 7, 2021

paul-tavares Sep 7, 2021

paul-tavares left a comment

kibanamachine commented Sep 7, 2021

kibanamachine commented Sep 7, 2021

[Security solution] [Endpoint] Fix bad artifact migration #111294

[Security solution] [Endpoint] Fix bad artifact migration #111294

Conversation

dasansol92 commented Sep 6, 2021 • edited Loading

Summary

For maintainers

dasansol92 commented Sep 7, 2021

elasticmachine commented Sep 7, 2021

paul-tavares Sep 7, 2021

Choose a reason for hiding this comment

elasticmachine commented Sep 7, 2021

paul-tavares left a comment

Choose a reason for hiding this comment

academo Sep 7, 2021

Choose a reason for hiding this comment

dasansol92 Sep 7, 2021

Choose a reason for hiding this comment

paul-tavares Sep 7, 2021

Choose a reason for hiding this comment

joshdover left a comment

Choose a reason for hiding this comment

paul-tavares Sep 7, 2021

Choose a reason for hiding this comment

paul-tavares Sep 7, 2021

Choose a reason for hiding this comment

paul-tavares left a comment

Choose a reason for hiding this comment

kibanamachine commented Sep 7, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Sep 7, 2021

💚 Backport successful

dasansol92 commented Sep 6, 2021 •

edited

Loading