[Security Solution] Resolver retrieve entity id of documents without field mapped #76562

jonathan-buttner · 2020-09-02T20:09:11Z

This PR fixes a bug in resolver and refactors a couple things:

Relaxes the query to ES in the entity route so that the document doesn't have to have entity_id field mapped
- Instead of having the ES query do the check that entity_id exists and is not empty, we use a runtime check to avoid the mapping issue
Refactors how timeline passes the indices to resolver
- Now timeline passes resolver all the indices (the ones the user has defined in the settings and the .siem-signals index)

screenshots

elasticmachine · 2020-09-02T20:09:14Z

Pinging @elastic/endpoint-app-team (Feature:Resolver)

elasticmachine · 2020-09-02T20:09:14Z

Pinging @elastic/endpoint-data-visibility-team (Team:Endpoint Data Visibility)

x-pack/plugins/security_solution/public/resolver/store/data/selectors.ts

x-pack/plugins/security_solution/public/resolver/view/clickthrough.test.tsx

x-pack/plugins/security_solution/public/timelines/components/graph_overlay/index.tsx

michaelolo24 · 2020-09-03T14:25:30Z

x-pack/plugins/security_solution/public/resolver/models/database_parameters.ts

+import _ from 'lodash';
+import { DatabaseParameters } from '../types';
+
+export function equal(param1: DatabaseParameters, param2?: DatabaseParameters) {


Why add this extra abstraction layer? For the type checking?

I think since timeline now sends resolver the ID and the indices we figured it'd be a good idea to combine them in a single object. Since indices is a string array we need some way to compare all the strings easily so Rob and I came up with this method. We might switch out the lodash equal though.

I was specifically talking about the abstraction of equal over isEqual. I understand the combined type, that makes sense. It's really more, if this equality check is going to be focused solely on checking the equality of DatabaseParameters, we know the shape, so might as well explicitly check the equality of the individual fields?

Oh you mean like, why not just use _.isEqual instead of our equal function? Or are you suggesting doing the comparisons directly in the places we're calling equal? I think @oatkiller was considering removing the lodash use so we'd have to implement an array comparison in that case. We'd probably want the array comparison at a minimum to be in a helper function because we do the DatabaseParameters check in like 3 different places I think.

I guess another option is we make DatabaseParameters into a class and move equal as a method of that class?

Yea, that was my thought, but since it's going to be something different from isEqual, then I would just leave it. Depending on what it is, it could still be more generalized into a util, but no need to do that here anyways

* change 'data' state shape, nesting the tree fetcher data * rename 'TreeFetcherParameters' from 'DatabaseParameters' to make it more specific to the API it works on * fix bug in 'equal' method of 'TreeFetcherParameters'` * use mockTreeFetcherParameters method in tests that need to specify a TreeFetcherParameters but when the value isn't relevant to the test * Hide Resolver if there is no databaseDocumentID * add doc comments

oatkiller · 2020-09-03T17:35:23Z

x-pack/plugins/security_solution/public/resolver/data_access_layer/factory.ts

@@ -38,13 +37,6 @@ export function dataAccessLayerFactory(
      });
    },

-    /**


The indices to use are now passed in to Resolver via props from the Security Solution app.

oatkiller · 2020-09-03T17:46:08Z

x-pack/plugins/security_solution/public/resolver/store/actions.ts

@@ -115,7 +115,7 @@ interface AppReceivedNewExternalProperties {
    /**
     * the `_id` of an ES document. This defines the origin of the Resolver graph.
     */
-    databaseDocumentID?: string;
+    databaseDocumentID: string;


This is required by ResolverProps now should it will always be present and can always be passed to this event.

oatkiller · 2020-09-03T17:46:34Z

x-pack/plugins/security_solution/public/resolver/store/data/action.ts

     */
-    databaseDocumentID: string;
+    parameters: TreeFetcherParameters;


instead of keeping track of just the databaseDocumentID used in the request, we now also keep track of the indices.

oatkiller · 2020-09-03T17:46:54Z

x-pack/plugins/security_solution/public/resolver/store/data/reducer.ts


 const initialState: DataState = {
  relatedEvents: new Map(),
  relatedEventsReady: new Map(),
  resolverComponentInstanceID: undefined,
+  tree: {},


The data related to fetching the resolver tree is now namespaced under tree.

I'll likely try to move this logic into a separate nested selector in the future.

oatkiller · 2020-09-03T17:48:11Z

x-pack/plugins/security_solution/public/resolver/store/data/reducer.ts

  } else if (action.type === 'appAbortedResolverDataRequest') {
-    if (action.payload === state.pendingRequestDatabaseDocumentID) {
+    if (treeFetcherParameters.equal(action.payload, state.tree.pendingRequestParameters)) {


Before we strictly compared the databaseDocumentIDs but now we use the equal method to compare the parameters.

oatkiller · 2020-09-03T17:54:18Z

x-pack/plugins/security_solution/public/resolver/view/use_state_syncing_actions.ts

 }) {
  const dispatch = useResolverDispatch();
  const locationSearch = useLocation().search;
  useLayoutEffect(() => {
    dispatch({
      type: 'appReceivedNewExternalProperties',
-      payload: { databaseDocumentID, resolverComponentInstanceID, locationSearch },
+      payload: { databaseDocumentID, resolverComponentInstanceID, locationSearch, indices },


We now keep the redux store in sync with the Resolver indices prop.

oatkiller · 2020-09-03T17:55:07Z

x-pack/plugins/security_solution/public/timelines/components/graph_overlay/index.tsx

@@ -137,6 +139,17 @@ const GraphOverlayComponent = ({
    globalFullScreen,
  ]);

+  const { signalIndexName } = useSignalIndex();
+  const [siemDefaultIndices] = useUiSetting$<string[]>(DEFAULT_INDEX_KEY);
+  const indices: string[] = useMemo(() => {


Concat the signal indices and the SIEM default indices and pass that to Resolver.

oatkiller · 2020-09-03T17:55:27Z

x-pack/plugins/security_solution/public/timelines/components/graph_overlay/index.tsx

-        databaseDocumentID={graphEventId}
-        resolverComponentInstanceID={currentTimeline.id}
-      />
+      {graphEventId !== undefined && (


Resolver cannot work without a databaseDocumentID so we hold off rendering it unless we have one.

Is <Suspense> still not ready yet?

I forgot that even existed.

oatkiller · 2020-09-03T17:56:35Z

x-pack/plugins/security_solution/public/timelines/components/graph_overlay/index.tsx

+        <StyledResolver
+          databaseDocumentID={graphEventId}
+          resolverComponentInstanceID={currentTimeline.id}
+          indices={indices}


This will render resolver with the SIEM default indices then once the signals index is fetched, it will rerender Resolver. This causes Resolver to start a request with just the default indices then cancel that request and re-request with the full set of indices.

@jonathan-buttner @XavierM Should we hold off rendering Resolver until the signals index has been fetched?

@andrew-goldstein Also worked on this.

I think it probably makes sense to wait for the response from the signals index api request 🤷

oatkiller · 2020-09-03T17:57:39Z

x-pack/plugins/security_solution/server/endpoint/routes/resolver/entity.ts

@@ -64,19 +59,6 @@ export function handleEntities(): RequestHandler<unknown, TypeOf<typeof validate
                    values: _id,
                  },
                },
-                {
-                  exists: {


By removing these checks, Resolver should work even if the new 7.9 index mappings aren't present. @jonathan-buttner Is that the right description?

Also, how should we mark this PR in release notes? Enhancement? or bug fix?

By removing these checks, Resolver should work even if the new 7.9 index mappings aren't present. @jonathan-buttner Is that the right description?

That's correct.

Also, how should we mark this PR in release notes? Enhancement? or bug fix?

@benskelker @spong This PR addresses the issue where resolver wouldn't render without the steps documented by Ben. Now resolver should render if the user upgrades their stack to 7.9.2 even if they don't perform the .siem-signals index mapping migration. Do we need to update release notes for this bug fix? If so how do I go about doing that?

Also to answer the original question I think it's probably a bug fix.

bkimmel · 2020-09-03T18:09:58Z

x-pack/plugins/security_solution/public/resolver/models/tree_fetcher_parameters.ts

+  if (first.length !== second.length) {
+    return false;
+  }
+  const firstSet = new Set(first);


ℹ️ It could be a WeakSet! It's your big chance WeakSet get in there! 🤣

haha I think to use a WeakSet the values would have to be objects though right? first is a string[] behind the scenes.

You're right 😢 next time, WeakSet

bkimmel · 2020-09-03T18:22:38Z

x-pack/plugins/security_solution/public/resolver/store/data/selectors.ts

 */
-export function resolverComponentInstanceID(state: DataState): string {
-  return state.resolverComponentInstanceID ? state.resolverComponentInstanceID : '';
+export function hadErrorLoadingTree(state: DataState): boolean {


❔ Suggest hadErrorLoadingLastTree to be more precise?

I'm not opposed to it. That being said, this code will only try once. If you get an error, you'll have it for good.

bkimmel

Nice work!

kibanamachine · 2020-09-03T22:57:36Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: e529bee

Build metrics

@kbn/optimizer bundle module count

id	value	diff	baseline
securitySolution	1940	+1	1939

async chunks size

id	value	diff	baseline
securitySolution	9.8MB	+8.5KB	9.8MB

History

💚 Build #72386 succeeded 97e14dd
💚 Build #72372 succeeded e97352c
💚 Build #72368 succeeded e0a62b3
💚 Build #72066 succeeded 1e37369
💔 Build #72038 failed 96392fa

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…field mapped (elastic#76562) * More comments * Adding tests for mapping without entity_id * Removing unnecessary comments * Fixing type errors * Removing unnecessary import * Fixups and style * change 'data' state shape, nesting the tree fetcher data * rename 'TreeFetcherParameters' from 'DatabaseParameters' to make it more specific to the API it works on * fix bug in 'equal' method of 'TreeFetcherParameters'` * use mockTreeFetcherParameters method in tests that need to specify a TreeFetcherParameters but when the value isn't relevant to the test * Hide Resolver if there is no databaseDocumentID * add doc comments * Fixing test name and adding comments * Pulling in roberts test name changes * [Resolver] Only render resolver once we have a signals index Co-authored-by: oatkiller <[email protected]>

…field mapped (#76562) (#76772) * More comments * Adding tests for mapping without entity_id * Removing unnecessary comments * Fixing type errors * Removing unnecessary import * Fixups and style * change 'data' state shape, nesting the tree fetcher data * rename 'TreeFetcherParameters' from 'DatabaseParameters' to make it more specific to the API it works on * fix bug in 'equal' method of 'TreeFetcherParameters'` * use mockTreeFetcherParameters method in tests that need to specify a TreeFetcherParameters but when the value isn't relevant to the test * Hide Resolver if there is no databaseDocumentID * add doc comments * Fixing test name and adding comments * Pulling in roberts test name changes * [Resolver] Only render resolver once we have a signals index Co-authored-by: oatkiller <[email protected]> Co-authored-by: oatkiller <[email protected]>

* master: (47 commits) Do not require id & description when creating a logstash pipeline (elastic#76616) Remove commented src/core/tsconfig file (elastic#76792) Replaced whitelistedHosts with allowedHosts in actions ascii docs (elastic#76731) [Dashboard First] Genericize Attribute Service (elastic#76057) [ci-metrics] unify distributable file count metrics (elastic#76448) [Security Solution][Detections] Handle conflicts on alert status update (elastic#75492) [eslint] convert to @typescript-eslint/no-unused-expressions (elastic#76471) [DOCS] Add default time range filter to advanced settings (elastic#76414) [Security Solution] Refactor NetworkTopNFlow to use Search Strategy (elastic#76249) [Dashboard] Update Index Patterns when Child Index Patterns Change (elastic#76356) [ML] Add option to Advanced Settings to set default time range filter for AD jobs (elastic#76347) Add CSM app to CODEOWNERS (elastic#76793) [Security Solution][Exceptions] - Updates exception item find sort field (elastic#76685) [Security Solution][Detections][Tech Debt] - Move to using common io-ts types (elastic#75009) [Lens] Drag dimension to replace (elastic#75895) URI encode the index names we fetch in the fetchIndices lib function. (elastic#76584) [Security Solution] Resolver retrieve entity id of documents without field mapped (elastic#76562) [Ingest Manager] validate agent route using AJV instead kbn-config-schema (elastic#76546) Updated non-dev usages of node-forge (elastic#76699) [Ingest Pipelines] Processor forms for processors K-S (elastic#75638) ...

jonathan-buttner added 2 commits September 2, 2020 13:48

More comments

3c3cba4

Adding tests for mapping without entity_id

e4c5153

jonathan-buttner added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:Endpoint Data Visibility Team managing the endpoint resolver Feature:Resolver Security Solution Resolver feature v7.9.2 labels Sep 2, 2020

jonathan-buttner requested review from oatkiller, bkimmel, michaelolo24, james-elastic and kqualters-elastic September 2, 2020 20:09

jonathan-buttner requested review from a team as code owners September 2, 2020 20:09

jonathan-buttner added the v7.10.0 label Sep 2, 2020

jonathan-buttner commented Sep 2, 2020

View reviewed changes

x-pack/plugins/security_solution/public/resolver/store/data/selectors.ts Outdated Show resolved Hide resolved

jonathan-buttner commented Sep 2, 2020

View reviewed changes

x-pack/plugins/security_solution/public/resolver/view/clickthrough.test.tsx Outdated Show resolved Hide resolved

jonathan-buttner added 2 commits September 2, 2020 16:18

Removing unnecessary comments

96392fa

Fixing type errors

b8af2f2

jonathan-buttner commented Sep 2, 2020

View reviewed changes

x-pack/plugins/security_solution/public/timelines/components/graph_overlay/index.tsx Outdated Show resolved Hide resolved

Removing unnecessary import

1e37369

michaelolo24 reviewed Sep 3, 2020

View reviewed changes

michaelolo24 approved these changes Sep 3, 2020

View reviewed changes

oatkiller and others added 3 commits September 3, 2020 13:04

Fixing test name and adding comments

e97352c

oatkiller reviewed Sep 3, 2020

View reviewed changes

bkimmel reviewed Sep 3, 2020

View reviewed changes

Pulling in roberts test name changes

97e14dd

bkimmel reviewed Sep 3, 2020

View reviewed changes

bkimmel approved these changes Sep 3, 2020

View reviewed changes

[Resolver] Only render resolver once we have a signals index

e529bee

jonathan-buttner merged commit ae093e5 into elastic:master Sep 4, 2020

jonathan-buttner deleted the entity-upgrade-fix branch September 4, 2020 13:24

jonathan-buttner mentioned this pull request Sep 4, 2020

[7.x] [Security Solution] Resolver retrieve entity id of documents without field mapped (#76562) #76772

Merged

jonathan-buttner removed the v7.9.2 label Sep 4, 2020

jonathan-buttner mentioned this pull request Sep 4, 2020

[Security Solution][Bug] Resolver handle entity id not being present in the signals mapping #76799

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Resolver retrieve entity id of documents without field mapped #76562

[Security Solution] Resolver retrieve entity id of documents without field mapped #76562

jonathan-buttner commented Sep 2, 2020 •

edited by oatkiller

Loading

elasticmachine commented Sep 2, 2020

elasticmachine commented Sep 2, 2020

michaelolo24 Sep 3, 2020

jonathan-buttner Sep 3, 2020

michaelolo24 Sep 3, 2020

jonathan-buttner Sep 3, 2020

michaelolo24 Sep 3, 2020

oatkiller Sep 3, 2020

oatkiller Sep 3, 2020

oatkiller Sep 3, 2020

oatkiller Sep 3, 2020

oatkiller Sep 3, 2020

oatkiller Sep 3, 2020

oatkiller Sep 3, 2020

oatkiller Sep 3, 2020

oatkiller Sep 3, 2020

bkimmel Sep 3, 2020

oatkiller Sep 3, 2020

oatkiller Sep 3, 2020

oatkiller Sep 3, 2020

jonathan-buttner Sep 3, 2020

oatkiller Sep 3, 2020

jonathan-buttner Sep 3, 2020

jonathan-buttner Sep 3, 2020

bkimmel Sep 3, 2020

jonathan-buttner Sep 3, 2020

bkimmel Sep 3, 2020

bkimmel Sep 3, 2020

oatkiller Sep 3, 2020

bkimmel left a comment

kibanamachine commented Sep 3, 2020

@@ @@ -38,13 +37,6 @@ export function dataAccessLayerFactory( @@
                     });
                   },
-                  /**

[Security Solution] Resolver retrieve entity id of documents without field mapped #76562

[Security Solution] Resolver retrieve entity id of documents without field mapped #76562

Conversation

jonathan-buttner commented Sep 2, 2020 • edited by oatkiller Loading

screenshots

elasticmachine commented Sep 2, 2020

elasticmachine commented Sep 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bkimmel left a comment

Choose a reason for hiding this comment

kibanamachine commented Sep 3, 2020

💚 Build Succeeded

Build metrics

@kbn/optimizer bundle module count

async chunks size

History

jonathan-buttner commented Sep 2, 2020 •

edited by oatkiller

Loading