[Security Solution] update endpoint list api to support united index #112758

joeypoon · 2021-09-21T22:11:25Z

Summary

metadata list api will now first attempt to query the united metadata index to save on extra fleet-agent queries.
if the index is not found, it will fallback to the existing query.
this PR is backwards compatible and will work even without endpoint package updates.

Checklist

Unit or functional tests were updated or added to match the most common scenarios

For maintainers

This was checked for breaking API changes and was labeled appropriately

joeypoon · 2021-09-21T22:13:19Z

x-pack/plugins/fleet/common/services/agent_status.ts

@@ -41,36 +41,42 @@ export function getAgentStatus(agent: Agent, now: number = Date.now()): AgentSta
  return 'online';
 }

-export function buildKueryForEnrollingAgents() {
-  return 'not (last_checkin:*)';
+export function buildKueryForEnrollingAgents(path: string = '') {


adding an optional path to the fields since we copy .fleet-agents mapping but are additionally nesting it

joeypoon · 2021-09-21T22:14:13Z

x-pack/plugins/security_solution/public/management/pages/endpoint_hosts/store/middleware.ts

-  async function fetchIndexPatterns(): Promise<IIndexPattern[]> {
+  // this needs to be called after endpointPackageVersion is loaded (getEndpointPackageInfo)
+  // or else wrong pattern might be loaded
+  async function fetchIndexPatterns(


use the endpoint package version to determine which index the kql bar should use for auto completion

joeypoon · 2021-09-21T22:15:20Z

x-pack/plugins/security_solution/server/endpoint/routes/metadata/handlers.ts

    return response.ok({
-      body: await mapToHostResultList(queryParams, hostListQueryResult, metadataRequestContext),
+      body: await legacyListMetadataQuery(


existing logic was just moved into this method with no change in logic.

joeypoon · 2021-09-21T22:15:45Z

x-pack/plugins/security_solution/server/endpoint/routes/metadata/handlers.ts

-
-    const result = await context.core.elasticsearch.client.asCurrentUser.search<HostMetadata>(
-      queryParams
+    const { unitedIndexExists, unitedQueryResponse } = await queryUnitedIndex(


this is the new query to the new united index

joeypoon · 2021-09-21T22:17:33Z

x-pack/plugins/security_solution/server/endpoint/routes/metadata/query_builders.ts

+  const filterIgnoredAgents =
+    ignoredAgentIds && ignoredAgentIds.length > 0
+      ? {
+          must_not: { terms: { 'agent.id': ignoredAgentIds } },


do we need to support HostDetails... like the legacy query here? if this is being written by endpoint I'm guessing it's possible they can run an older version which still uses HostDetails?

elasticmachine · 2021-09-22T14:48:44Z

Pinging @elastic/security-onboarding-and-lifecycle-mgt (Team:Onboarding and Lifecycle Mgt)

joeypoon · 2021-09-22T18:52:36Z

@elasticmachine merge upstream

elasticmachine · 2021-09-22T18:52:47Z

Pinging @elastic/fleet (Team:Fleet)

nchaulet

Fleet changes 🚀 (these strings interpolation seems a little fragile to me and we should probably move to another pattern like a query builder or other one day to have something more robust here, at least we have some functional tests that cover these queries)

kevinlog · 2021-09-23T19:01:49Z

@joeypoon - awesome job on this so far.

I check it out, tried it, and found a few issues. Let me know if you have any questions.

Here are the cases I tested.

Original/backwards compatible mode

Enroll a single real Agent with Endpoint, Endpoint shows in Endpoint list ✅
Unenroll Agent, Endpoint is removed from Endpoint list ✅
Enroll Agent with Endpoint, Endpoint appears in list, remove Endpoint integration ONLY, Endpoint is removed from list ✅
Generate 100 docs with the generator, all docs appear in list ✅

New transforms

Enroll a single real Agent with Endpoint, Endpoint shows in Endpoint list ✅
Unenroll Agent, Endpoint is removed from Endpoint list ❌
- The Endpoint record remained in the list while the Agent was removed from the Agents list
Enroll Agent with Endpoint, Endpoint appears in list, remove Endpoint integration ONLY, Endpoint is removed from list ✅
- I tried this scenario a few times. I added the Endpoint integration to the Agent policy and deleted it a handful of times. It worked as expected each time
Generate 100 docs with the generator, all docs appear in list ❌ ❓
- I saw some odd behavior on this one. I could see the docs coming into the destination indices, but the Endpoint list didn't render correctly. I think the generator may need to be updated? There are some screenshots below of the scenario.

You can see pagination and "Showing" number, but it doesn't render.

I can see the docs in the indices, so may it has to do with the shape of them?

joeypoon · 2021-09-23T21:39:03Z

thanks for checking @kevinlog

Unenroll Agent, Endpoint is removed from Endpoint list ❌

latest push should fix that one and also fixes the issue with generated docs kind of showing up in the list (they don't show up at all now, this is expected since generated docs don't currently merge in the endpoint metadata into the doc).

…ta index metadata list api will now first attempt to query the united metadata index to save on extra fleet-agent queries. if the index is not found, it will fallback to the existing query.

joeypoon · 2021-09-27T17:04:21Z

endpoint generator also works with united index with latest commit 👌

pzl

🐑 ?

kibanamachine · 2021-09-27T18:55:41Z

💚 Build Succeeded

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	2224	2225	+1

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	4.2MB	4.2MB	+231.0B

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`fleet`	132.4KB	132.3KB	-15.0B
`securitySolution`	108.4KB	108.4KB	+80.0B
total			+65.0B

History

💔 Build #155506 failed 6730ea8a7180508c7d6217c3561ced669bf6bc67
💔 Build #155501 failed 7fea80600d02ac554a00165335146297a3cc70c5
💚 Build #155084 succeeded 822942ddc8658a83c182f897f4cc63a5d72df496
💔 Build #154997 failed 1ee59e2e1b5d3e72c99d4dd48a31179992780b20
💔 Build #154790 failed 0f95c98a8b68f5298564735f01b11942416b306b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…lastic#112758)

kibanamachine · 2021-09-27T19:54:14Z

💚 Backport successful

Status	Branch	Result
✅	7.x

This backport PR will be merged automatically after passing CI.

…112758) (#113175) Co-authored-by: Joey F. Poon <[email protected]>

joeypoon added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution auto-backport Deprecated - use backport:version if exact versions are needed v7.16.0 labels Sep 21, 2021

joeypoon requested a review from a team September 21, 2021 22:11

joeypoon commented Sep 21, 2021

View reviewed changes

joeypoon force-pushed the feature/united-transform branch 2 times, most recently from 0f95c98 to 1ee59e2 Compare September 22, 2021 14:48

joeypoon marked this pull request as ready for review September 22, 2021 14:48

joeypoon requested review from a team as code owners September 22, 2021 14:48

joeypoon requested review from pzl and paul-tavares September 22, 2021 14:48

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Sep 22, 2021

nchaulet approved these changes Sep 22, 2021

View reviewed changes

joeypoon force-pushed the feature/united-transform branch 2 times, most recently from 7fea806 to 6730ea8 Compare September 23, 2021 21:30

joeypoon added 3 commits September 27, 2021 11:50

[Security Solution] update endpoint list api to support united metada…

f84dc52

…ta index metadata list api will now first attempt to query the united metadata index to save on extra fleet-agent queries. if the index is not found, it will fallback to the existing query.

filter out unenrolled agents

0028c7f

update endpoint generator for united index + fix tests

c8e6451

joeypoon force-pushed the feature/united-transform branch from 6730ea8 to c8e6451 Compare September 27, 2021 16:51

pzl approved these changes Sep 27, 2021

View reviewed changes

joeypoon merged commit 94e7844 into elastic:master Sep 27, 2021

joeypoon deleted the feature/united-transform branch September 27, 2021 19:51

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Sep 27, 2021

[Security Solution] update endpoint list api to support united index (e…

de80867

…lastic#112758)

kibanamachine mentioned this pull request Sep 27, 2021

[7.x] [Security Solution] update endpoint list api to support united index (#112758) #113175

Merged

kibanamachine added a commit that referenced this pull request Sep 27, 2021

[Security Solution] update endpoint list api to support united index (#…

ff8bf51

…112758) (#113175) Co-authored-by: Joey F. Poon <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] update endpoint list api to support united index #112758

[Security Solution] update endpoint list api to support united index #112758

joeypoon commented Sep 21, 2021

joeypoon Sep 21, 2021

joeypoon Sep 21, 2021

joeypoon Sep 21, 2021

joeypoon Sep 21, 2021

joeypoon Sep 21, 2021

elasticmachine commented Sep 22, 2021

joeypoon commented Sep 22, 2021

elasticmachine commented Sep 22, 2021

nchaulet left a comment

kevinlog commented Sep 23, 2021

joeypoon commented Sep 23, 2021 •

edited

Loading

joeypoon commented Sep 27, 2021

pzl left a comment

kibanamachine commented Sep 27, 2021

kibanamachine commented Sep 27, 2021

[Security Solution] update endpoint list api to support united index #112758

[Security Solution] update endpoint list api to support united index #112758

Conversation

joeypoon commented Sep 21, 2021

Summary

Checklist

For maintainers

joeypoon Sep 21, 2021

Choose a reason for hiding this comment

joeypoon Sep 21, 2021

Choose a reason for hiding this comment

joeypoon Sep 21, 2021

Choose a reason for hiding this comment

joeypoon Sep 21, 2021

Choose a reason for hiding this comment

joeypoon Sep 21, 2021

Choose a reason for hiding this comment

elasticmachine commented Sep 22, 2021

joeypoon commented Sep 22, 2021

elasticmachine commented Sep 22, 2021

nchaulet left a comment

Choose a reason for hiding this comment

kevinlog commented Sep 23, 2021

joeypoon commented Sep 23, 2021 • edited Loading

joeypoon commented Sep 27, 2021

pzl left a comment

Choose a reason for hiding this comment

kibanamachine commented Sep 27, 2021

💚 Build Succeeded

Metrics [docs]

Module Count

Async chunks

Page load bundle

History

kibanamachine commented Sep 27, 2021

💚 Backport successful

joeypoon commented Sep 23, 2021 •

edited

Loading