[Endpoint] Adding a Resolver component that lets the user click and drag to pan and pinch to zoom #53619

oatkiller · 2019-12-19T18:20:39Z

Summary

Resolver is a map. It shows processes that ran on a computer. The processes are drawn as nodes and lines connect processes with their parents.

Resolver is not yet implemented in Kibana. This PR adds a 'map' type UX. The user can click and drag to pan the map and zoom using trackpad pinching (or ctrl and mousewheel.)

There is no code providing actual data. Sample data is included. The sample data is used to draw a map. The fundamental info needed is:

process names
the parent of a process

With this info we can topologically lay out the processes. The sample data isn't yet in a realistic format. We'll be fixing that soon.

Related issue: https://github.com/elastic/endpoint-app-team/issues/30

This does not implement accessibility features. Those will be added: https://github.com/elastic/endpoint-app-team/issues/40 and https://github.com/elastic/endpoint-app-team/issues/34
This embeddable is not released or available by default, so this should be fine for now.

We aren't adding documentation yet as this feature has not been developed enough.

Checklist

Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR.

This was checked for cross-browser compatibility, including a check against IE11
~~Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~Documentation was added for features that require explanation or tutorials~~
Unit or functional tests were updated or added to match the most common scenarios
~~This was checked for keyboard-only and screenreader accessibility~~

For maintainers

This was checked for breaking API changes and was labeled appropriately
This includes a feature addition or change that requires a release note and was labeled appropriately

oatkiller · 2019-12-20T15:00:58Z

x-pack/plugins/endpoint/public/embeddables/resolver/view/index.tsx

+  return [rect, ref];
+}
+
+const DiagnosticDot = styled(


@bkimmel here's where a component can find its position relative to the parent resolver component

elasticmachine · 2019-12-20T22:41:11Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 1bcdbde2f117c56638dcabd22dd470f7a47a26c7

History

💔 Build #16992 failed b0c6385b7cb447211ec189e63380ffc604c74a3d
💔 Build #16780 failed 85d987c14c905cc30ceb5e8ed2e0dd64e3e489d0
💔 Build #16766 failed 61d8ec29727bb7368cf552061eee7892fa2027c4
💔 Build #16749 failed c0054d9987643a942c7f79ffc870ac7b673ff4d1

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

oatkiller · 2020-01-09T19:25:53Z

x-pack/plugins/endpoint/package.json

+  "license": "Elastic-License",
+  "scripts": {},
+  "dependencies": {
+    "react-redux": "^7.1.0"


We depend on react-redux's hooks.

bkimmel · 2020-01-09T21:16:28Z

x-pack/test/plugin_functional/plugins/resolver_test/public/applications/resolver_test/index.tsx

+      const [renderTarget, setRenderTarget] = React.useState<HTMLDivElement | null>(null);
+
+      useEffect(() => {
+        let cleanUp;


❔ If you're using cleanUp for something here, it is really not clear what. I see it gets the reject side of the Promise below, but I don't understand where or how it's used at all. Can you explain?

I added comments to explain but i'll reiterate here. The 'effect' is responsible for waiting for a promise to resolve and then setting component 'state' from that resolved value.

There is complexity because the promise comes in as a React prop. The prop could change at any point. When the prop changes, we call Promise.race. If the value is resolved, we set the value in state. If the prop changes, we cancel the race by rejecting.

package.json

@@ -99,6 +99,7 @@
      "x-pack",
      "x-pack/plugins/*",
      "x-pack/legacy/plugins/*",
+      "x-pack/plugins/endpoint",


andrewstucki · 2020-01-13T19:15:32Z

x-pack/plugins/endpoint/public/embeddables/resolver/lib/matrix3.ts

+/**
+ * Return a new matrix which is the sum of the two passed in.
+ */
+export function add(


Since all of these seem to have name collisions between matrix v. vector addition, multiplication, etc. wondering if it makes sense to nest this export under some sort of class or named constant. Just want to avoid littering code with things like:

import { add as matrixAdd } from 'matrix3' import { add as vectorAdd } from 'vector2'

as well as having to back reference whether we're operating on matrices, vectors, lists, etc. because we've lost all context of what add we're working with at that point...

a module is already a logical grouping of exports. you can:

import * as matrix3 from 'matrix3' // ... matrix3.add(a,b)

if ya want. also you can get context from your editor regardless of how it was imported. In this case it was imported as addMatrix:

andrewstucki · 2020-01-13T19:17:22Z

x-pack/plugins/endpoint/public/embeddables/resolver/models/process_event.ts

+    data_buffer: { event_type_full: type, event_subtype_full: subType },
+  } = event;
+
+  if (type === 'process_event') {


it would be nice to have this stuff scoped to particular transforms for endgame v0 events, mainly because none of this stuff is ECS compatible as it stands at the moment

This will change once we use real, ECS compatible data, as opposed to sample data. We will hopefully do this soon.

@andrewstucki I've never been able to get the streaming feature to produce a set of events that could be used for resolver. in all of my attempts, enough events are missing that I can't render a resolver graph from it. If you could provide some more realistic sample data we'll use it. Otherwise we can wait for @jonathan-buttner to finish a Resolver data generator. @jonathan-buttner is that ready?

So, we should sit down together and see what you actually need, because the events shipped have literally everything in them that are part of our raw events--so if there's not enough data, we have a problem, because it means that the endpoint itself isn't shipping data compatible with what you're trying to visualize.

sorry for the confusing explanation. the events have enough data, I just can't get the streaming feature to send all of the events.

@oatkiller the data generator is close but still working out some issues. We haven't started on a backend implementation yet but that's next on the list.

andrewstucki · 2020-01-13T19:22:12Z

x-pack/plugins/endpoint/public/embeddables/resolver/models/process_event.ts

+}
+
+export function uniqueParentPidForProcess(event: ProcessEvent) {
+  return event.data_buffer.source_id;


I'd be very careful with nomenclature here, because this doesn't look like an "event" to me--"event"s are our raw data sources, for which we don't have a "source_id" field. This looks more like some sort of investigation or alert, something that originally came from the hunt_data_schema--if that's true then the applications for resolver out of the box are going to be very limited. Instead this should be working on the raw event data if possible. In which case:

v0 (data coming from the SMP) will be looking at the field endgame.unique_ppid for process events

v1 (data coming from Elastic-native sensor) will be looking at the field endpoint.process.parent.entity_id for events with event.category == 'process'

Let's sync on all of this. But basically, this is using some random sample data that has pids, unique ppids, and process names. that's all that we need to test the layout logic. Once we get a BE, we'll get this stuff working against real data.

andrewstucki · 2020-01-13T19:23:23Z

x-pack/plugins/endpoint/public/embeddables/resolver/models/process_event.ts

+}
+
+export function uniquePidForProcess(event: ProcessEvent) {
+  return event.data_buffer.node_id;


see comment about source_id stuff:

v0 -- endgame.unique_pid

v1 -- endpoint.process.entity_id

I'm not totally sure how types will work with events, but I image having a single Event type which encompasses all of the events that we'll ever deal with. When we change that type, ts will help us fix all the code that references it. If we get sample data that has the right format, we'll change the type and all of the code then.

andrewstucki · 2020-01-13T19:24:19Z

x-pack/plugins/endpoint/public/embeddables/resolver/models/process_event_test_helpers.ts

+
+export function mockProcessEvent(
+  parts: {
+    data_buffer: { node_id: ProcessEvent['data_buffer']['node_id'] };


See comments about what a "process event" is -- none of these fields exist in process events that we ship today to Elasticsearch or in the events we'll ship in the future.

Resolver is a map. It shows processes that ran on a computer. The processes are drawn as nodes and lines connect processes with their parents. Resolver is not yet implemented in Kibana. This PR adds a 'map' type UX. The user can click and drag to pan the map and zoom using trackpad pinching (or ctrl and mousewheel.) There is no code providing actual data. Sample data is included. The sample data is used to draw a map. The fundamental info needed is: process names the parent of a process With this info we can topologically lay out the processes. The sample data isn't yet in a realistic format. We'll be fixing that soon. Related issue: elastic/endpoint-app-team#30

kibanamachine · 2020-02-18T14:13:21Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: b5df646

History

💚 Build #20250 succeeded b5df646
💚 Build #20114 succeeded 9d61b26a4f391e8c3155e604a4e48bd15f6d19ee
💚 Build #19934 succeeded 0c5c9cdb781eb77361631722107f7fd136e54502
💚 Build #19923 succeeded 78faed0b1a4f0bdf1548fb93bb42286b51b1dd2e
💔 Build #19884 failed eb1521c84ea983dfed91af971baf9e8c9adc6f2e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* Add Endpoint plugin and Resolver embeddable (#51994) * Add functional tests for plugins to x-pack (so we can do a functional test of the Resolver embeddable) * Add Endpoint plugin * Add Resolver embeddable * Test that Resolver embeddable can be rendered Conflicts: x-pack/.i18nrc.json x-pack/test/api_integration/apis/index.js * [Endpoint] Register endpoint app (#53527) * register app, create functional test * formatting * update tests * adjust test data for endpoint * add endpoint tests for testing spaces, app enabled, disabled, etc * linting * add read privileges to endpoint * rename variable since its used now * remove deprecated context * remove unused variable * fix type check * correct test suite message Co-Authored-By: Larry Gregory <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Larry Gregory <[email protected]> * [Endpoint] add react router to endpoint app (#53808) * add react router to endpoint app * linting * linting * linting * correct tests * change history from hash to browser, add new test util * remove default values in helper functions * fix type check, use FunctionComponent as oppsed to FC * use BrowserRouter component * use BrowserRouter component lin * add comments to test framework, change function name to include browserHistory Co-authored-by: Elastic Machine <[email protected]> * EMT-issue-65: add endpoint list api (#53861) add endpoint list api * EMT-65:always return accurate endpoint count (#54423) EMT-65:always return accurate endpoint count, independent of paging properties * Resolver component w/ sample data (#53619) Resolver is a map. It shows processes that ran on a computer. The processes are drawn as nodes and lines connect processes with their parents. Resolver is not yet implemented in Kibana. This PR adds a 'map' type UX. The user can click and drag to pan the map and zoom using trackpad pinching (or ctrl and mousewheel.) There is no code providing actual data. Sample data is included. The sample data is used to draw a map. The fundamental info needed is: process names the parent of a process With this info we can topologically lay out the processes. The sample data isn't yet in a realistic format. We'll be fixing that soon. Related issue: elastic/endpoint-app-team#30 * Resolver test plugin not using mount context. (#54933) Mount context was deprecated. Use core.getStartServices() instead. * Resolver nonlinear zoom (#54936) * [Endpoint] add Redux saga Middleware and app Store (#53906) * Added saga library * Initialize endpoint app redux store * Resolver is overflow: hidden to prevent obscured elements from showing up (#55076) * [Endpoint] Fix saga to start only after store is created and stopped on app unmount (#55245) - added `stop()`/`start()` methods to the Saga Middleware creator factory - adjust tests based on changes - changed application `renderApp` to stop sagas when react app is unmounted * Resolver zoom, pan, and center controls (#55221) * Resolver zoom, pan, and center controls * add tests, fix north panning * fix type issue * update west and east panning to behave like google maps * [Endpoint] FIX: Increase tests `sleep` default duration back to 100ms (#55492) Revert `sleep()` default duration, in the saga tests, back to 100ms in order to prevent intermittent failures during CI runs. Fixes #55464 Fixes #55465 * [Endpoint] EMT-65: make endpoint data types common, restructure (#54772) [Endpoint] EMT-65: make endpoint data types common, use schema changes * Basic Functionality Alert List (#55800) * sets up initial grid and data type * data feeds in from backend but doesnt update * sample data feeding in correctly * Fix combineReducers issue by importing Redux type from 'redux' package * Add usePageId hook that fires action when user navigates to page * Strict typing for middleware * addresses comments and uses better types * move types to common/types.ts * Move types to endpoint/types.ts, address PR comments blah 2 Co-authored-by: Pedro Jaramillo <[email protected]> * [Endpoint] Add Endpoint Details route (#55746) * Add Endpoint Details route * add Endpoint Details tests * sacrifices to the Type gods * update to latest endpoint schema Co-authored-by: Elastic Machine <[email protected]> * [Endpoint] EMT-67: add kql support for endpoint list (#56328) [Endpoint] EMT-67: add kql support for endpoint list * [Endpoint] ERT-82 ERT-83 ERT-84: Alert list API with pagination (#56538) * ERT-82 ERT-83 ERT-84 (partial): Add Alert List API with pagination * Better type safety for alert list API * Add Test to Verify Endpoint App Landing Page (#57129) Conflicts: x-pack/test/functional/page_objects/index.ts * fixes render bug in alert list (#57152) Co-authored-by: Elastic Machine <[email protected]> * Resolver: Animate camera, add sidebar (#55590) This PR adds a sidebar navigation. clicking the icons in the nav will focus the camera on the different nodes. There is an animation effect when the camera moves. Conflicts: yarn.lock * [Endpoint] Task/basic endpoint list (#55623) * Adds host management list to endpoint security plugin Co-authored-by: Elastic Machine <[email protected]> * [Endpoint] Policy List UI route and initial view (#56918) * Initial Policy List view * Add `endpoint/policy` route and displays Policy List * test cases (both unit and functional) Does not yet interact with API (Ingest). * Add ApplicationService app status management (#50223) This was already backported, but changes to endpoint app could not be backported, since endpoint app itself hadn't been backported. Now that the endpoint app is backported, reapply the endpoint specific changes from the original commit. * Implements `getStartServices` on server-side (#55156) This was already backported, but changes to endpoint app could not be backported, since endpoint app itself hadn't been backported. Now that the endpoint app is backported, reapply the endpoint specific changes from the original commit. * [ui/utils/query_string]: Remove unused methods & migrate apps to querystring lib (#56957) This was already backported, but changes to endpoint app could not be backported, since endpoint app itself hadn't been backported. Now that the endpoint app is backported, reapply the endpoint specific changes from the original commit. Co-authored-by: Kevin Logan <[email protected]> Co-authored-by: Elastic Machine <[email protected]> Co-authored-by: Larry Gregory <[email protected]> Co-authored-by: nnamdifrankie <[email protected]> Co-authored-by: Davis Plumlee <[email protected]> Co-authored-by: Paul Tavares <[email protected]> Co-authored-by: Pedro Jaramillo <[email protected]> Co-authored-by: Dan Panzarella <[email protected]> Co-authored-by: Madison Caldwell <[email protected]> Co-authored-by: Charlie Pichette <[email protected]> Co-authored-by: Candace Park <[email protected]> Co-authored-by: Pierre Gayvallet <[email protected]> Co-authored-by: Alexey Antonov <[email protected]>

oatkiller force-pushed the resolver-harness-without-endpoint-app-pt2 branch from c0054d9 to 61d8ec2 Compare December 19, 2019 19:20

oatkiller commented Dec 20, 2019

View reviewed changes

oatkiller force-pushed the resolver-harness-without-endpoint-app-pt2 branch from 85d987c to b0c6385 Compare December 20, 2019 18:59

oatkiller force-pushed the resolver-harness-without-endpoint-app-pt2 branch 7 times, most recently from 26cac31 to 0e4a5a6 Compare January 9, 2020 17:13

peluja1012 approved these changes Jan 9, 2020

View reviewed changes

james-elastic added the Team:Endpoint Data Visibility Team managing the endpoint resolver label Jan 9, 2020

oatkiller commented Jan 9, 2020

View reviewed changes

oatkiller force-pushed the resolver-harness-without-endpoint-app-pt2 branch from c4ce449 to a02a0d4 Compare January 9, 2020 19:52

bkimmel reviewed Jan 9, 2020

View reviewed changes

peluja1012 mentioned this pull request Jan 13, 2020

Basic layout process nodes and edges oatkiller/kibana#1

Closed

7 tasks

oatkiller force-pushed the resolver-harness-without-endpoint-app-pt2 branch from 608d94f to eb1521c Compare January 13, 2020 18:57

andrewstucki reviewed Jan 13, 2020

View reviewed changes

oatkiller force-pushed the resolver-harness-without-endpoint-app-pt2 branch 2 times, most recently from 2ed5a47 to 9d61b26 Compare January 14, 2020 13:22

elastic deleted a comment from peluja1012 Jan 14, 2020

oatkiller added Feature:Resolver Security Solution Resolver feature release_note:skip Skip the PR/issue when compiling release notes v7.6.0 v8.0.0 labels Jan 14, 2020

elastic deleted a comment from peluja1012 Jan 14, 2020

oatkiller and others added 20 commits January 14, 2020 12:58

we are cool

a438da6

megafactor

261e0f1

refactor edge lines

3d89e00

cleanup

ee2b076

loop unravel

7b241ae

cleanup

bc26420

cleanup

b2d8d5b

more elite code

bc840fc

more stuff

2ae6307

just about good

d78c475

we must never speak of this

5f37888

get isometric

5112ed7

remove todo

1ebcddc

update snapshots to match isometric stuff

1fef3b7

remove unused function

679fa35

add some comments

57fa7f9

more commints

2462aa2

do comments right

015c110

comments

ce7880b

remove endpoint workspace

b5df646

oatkiller force-pushed the resolver-harness-without-endpoint-app-pt2 branch from 9d61b26 to b5df646 Compare January 14, 2020 17:58

davisplumlee approved these changes Jan 14, 2020

View reviewed changes

kqualters-elastic approved these changes Jan 14, 2020

View reviewed changes

oatkiller merged commit 387da98 into elastic:master Jan 14, 2020

oatkiller deleted the resolver-harness-without-endpoint-app-pt2 branch January 14, 2020 20:38

kevinlog added the Feature:Endpoint Elastic Endpoint feature label Feb 18, 2020

kevinlog changed the title ~~Adding a Resolver component that lets the user click and drag to pan and pinch to zoom~~ [Endpoint] Adding a Resolver component that lets the user click and drag to pan and pinch to zoom Feb 18, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Endpoint] Adding a Resolver component that lets the user click and drag to pan and pinch to zoom #53619

[Endpoint] Adding a Resolver component that lets the user click and drag to pan and pinch to zoom #53619

oatkiller commented Dec 19, 2019 •

edited

Loading

oatkiller Dec 20, 2019

elasticmachine commented Dec 20, 2019

oatkiller Jan 9, 2020

bkimmel Jan 9, 2020

oatkiller Jan 14, 2020

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

andrewstucki Jan 13, 2020

oatkiller Jan 13, 2020 •

edited

Loading

andrewstucki Jan 13, 2020

peluja1012 Jan 13, 2020

oatkiller Jan 13, 2020

andrewstucki Jan 13, 2020

oatkiller Jan 14, 2020

jonathan-buttner Jan 14, 2020

andrewstucki Jan 13, 2020

oatkiller Jan 13, 2020

andrewstucki Jan 13, 2020

oatkiller Jan 13, 2020

andrewstucki Jan 13, 2020

kibanamachine commented Feb 18, 2020

[Endpoint] Adding a Resolver component that lets the user click and drag to pan and pinch to zoom #53619

[Endpoint] Adding a Resolver component that lets the user click and drag to pan and pinch to zoom #53619

Conversation

oatkiller commented Dec 19, 2019 • edited Loading

Summary

Checklist

For maintainers

Choose a reason for hiding this comment

elasticmachine commented Dec 20, 2019

💔 Build Failed

History

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

Choose a reason for hiding this comment

oatkiller Jan 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Feb 18, 2020

💔 Build Failed

History

oatkiller commented Dec 19, 2019 •

edited

Loading

oatkiller Jan 13, 2020 •

edited

Loading