Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Endpoint] Recursive resolver children #61914

Merged
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
42 commits
Select commit Hold shift + click to select a range
42e9f93
Add child recursion to resolve multiple levels of children
Feb 20, 2020
228ed62
Fix comment
Feb 20, 2020
d3d82a7
Update with in-memory tree construction
Feb 21, 2020
4d248bc
Refactor fetcher
Feb 22, 2020
58009e5
Fix up tests
Feb 24, 2020
3b35b1a
Fix unit tests
Feb 24, 2020
a36ef94
Add stats resolution for all nodes
Feb 24, 2020
63ad7eb
fix stats field and throw in boilerplate for alerts
Feb 24, 2020
2b4ce9f
Fix ordering issue
Feb 24, 2020
d686f17
Merge branch 'master' into recursive-resolver-children
elasticmachine Feb 24, 2020
0731e88
Fix unit test
Feb 24, 2020
dd4ac56
Get rid of 'any' by using keyof
Feb 25, 2020
51fcea6
Merge branch 'master' of https://github.com/elastic/kibana into recur…
Feb 27, 2020
35d0631
Merge branch 'master' into recursive-resolver-children
elasticmachine Feb 27, 2020
b4a5a6a
Merge branch 'master' of https://github.com/elastic/kibana into recur…
Mar 2, 2020
8899993
Merge branch 'master' into recursive-resolver-children
kqualters-elastic Mar 25, 2020
c4bcbbb
Merge branch 'master' into recursive-resolver-children
kqualters-elastic Mar 26, 2020
50d5d35
[Endpoint] single resolver api
kqualters-elastic Mar 30, 2020
e5e1c96
Add error handling to reducer/selector
kqualters-elastic Mar 31, 2020
1d6cd24
Use node type for children
kqualters-elastic Mar 31, 2020
e50dc02
Merge branch 'master' into recursive-resolver-children
kqualters-elastic Mar 31, 2020
139c4f2
Comments. Remove magic numbers. change layout a bit. align node with …
Apr 1, 2020
7fdf029
WIP, not working, not good
Apr 1, 2020
3e9fa09
Revert "WIP, not working, not good"
Apr 1, 2020
63f13a7
fix layout bug
Apr 1, 2020
0f5963d
demo hacks
Apr 1, 2020
307318d
change the viewbox so it doesn't overlap with others. change the vert…
Apr 1, 2020
eed8ca3
Return types, use shared event model
kqualters-elastic Apr 8, 2020
e5a3a48
WIP, no ancestors
kqualters-elastic Apr 9, 2020
b822069
Merge branch 'master' into recursive-resolver-children
kqualters-elastic Apr 14, 2020
288aadd
Merge remote-tracking branch 'oatkiller/recursive-resolver-children' …
kqualters-elastic Apr 15, 2020
24abb1f
Fixing ancestors
jonathan-buttner Apr 15, 2020
8321e99
Use ancestors
kqualters-elastic Apr 20, 2020
9b9f46e
Merge branch 'master' into recursive-resolver-children
kqualters-elastic Apr 21, 2020
7f47791
Change resolver tree to use a map for caching already queried process…
kqualters-elastic Apr 24, 2020
8ebf221
Revert defs.tsx changes
kqualters-elastic Apr 24, 2020
a607144
Remove test looking for non existant event field
kqualters-elastic Apr 24, 2020
b47f270
Address pr comments
kqualters-elastic Apr 27, 2020
b430f18
Merge remote-tracking branch 'upstream/master' into recursive-resolve…
kqualters-elastic Apr 28, 2020
5725da1
Remove comments linking to internal urls
kqualters-elastic Apr 28, 2020
3ae0c85
Import index pattern typedef from correct location
kqualters-elastic Apr 28, 2020
8262af9
Merge branch 'master' into recursive-resolver-children
kqualters-elastic Apr 28, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 14 additions & 1 deletion x-pack/plugins/endpoint/common/generate_data.ts
Original file line number Diff line number Diff line change
Expand Up @@ -307,7 +307,7 @@ export class EndpointDocGenerator {
process: {
entity_id: options.entityID ? options.entityID : this.randomString(10),
parent: options.parentEntityID ? { entity_id: options.parentEntityID } : undefined,
name: options.processName ? options.processName : 'powershell.exe',
name: options.processName ? options.processName : randomProcessName(),
},
};
}
Expand Down Expand Up @@ -645,3 +645,16 @@ export class EndpointDocGenerator {
return uuid.v4({ random: [...this.randomNGenerator(255, 16)] });
}
}

const fakeProcessNames = [
'lsass.exe',
'notepad.exe',
'mimikatz.exe',
'powershell.exe',
'iexlorer.exe',
'explorer.exe',
];
/** Return a random fake process name */
function randomProcessName(): string {
return fakeProcessNames[Math.floor(Math.random() * fakeProcessNames.length)];
}
33 changes: 25 additions & 8 deletions x-pack/plugins/endpoint/common/models/event.ts
Original file line number Diff line number Diff line change
Expand Up @@ -4,28 +4,45 @@
* you may not use this file except in compliance with the Elastic License.
*/

import { EndpointEvent, LegacyEndpointEvent } from '../types';
import { LegacyEndpointEvent, ResolverEvent } from '../types';

export function isLegacyEvent(
event: EndpointEvent | LegacyEndpointEvent
): event is LegacyEndpointEvent {
export function isLegacyEvent(event: ResolverEvent): event is LegacyEndpointEvent {
return (event as LegacyEndpointEvent).endgame !== undefined;
}

export function eventTimestamp(
event: EndpointEvent | LegacyEndpointEvent
): string | undefined | number {
export function eventTimestamp(event: ResolverEvent): string | undefined | number {
if (isLegacyEvent(event)) {
return event.endgame.timestamp_utc;
} else {
return event['@timestamp'];
}
}

export function eventName(event: EndpointEvent | LegacyEndpointEvent): string {
export function eventName(event: ResolverEvent): string {
if (isLegacyEvent(event)) {
return event.endgame.process_name ? event.endgame.process_name : '';
} else {
return event.process.name;
}
}

export function eventId(event: ResolverEvent): string {
if (isLegacyEvent(event)) {
return event.endgame.serial_event_id ? String(event.endgame.serial_event_id) : '';
}
return event.event.id;
}

export function entityId(event: ResolverEvent): string {
if (isLegacyEvent(event)) {
return event.endgame.unique_pid ? String(event.endgame.unique_pid) : '';
}
return event.process.entity_id;
}

export function parentEntityId(event: ResolverEvent): string | undefined {
if (isLegacyEvent(event)) {
return event.endgame.unique_ppid ? String(event.endgame.unique_ppid) : undefined;
}
return event.process.parent?.entity_id;
}
59 changes: 59 additions & 0 deletions x-pack/plugins/endpoint/common/schema/resolver.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/

import { schema } from '@kbn/config-schema';

/**
* Used to validate GET requests for a complete resolver tree.
*/
export const validateTree = {
params: schema.object({ id: schema.string() }),
query: schema.object({
children: schema.number({ defaultValue: 10, min: 0, max: 100 }),
generations: schema.number({ defaultValue: 3, min: 0, max: 3 }),
ancestors: schema.number({ defaultValue: 3, min: 0, max: 5 }),
events: schema.number({ defaultValue: 100, min: 0, max: 1000 }),
afterEvent: schema.maybe(schema.string()),
afterChild: schema.maybe(schema.string()),
legacyEndpointID: schema.maybe(schema.string()),
}),
};

/**
* Used to validate GET requests for non process events for a specific event.
*/
export const validateEvents = {
params: schema.object({ id: schema.string() }),
query: schema.object({
events: schema.number({ defaultValue: 100, min: 1, max: 1000 }),
afterEvent: schema.maybe(schema.string()),
legacyEndpointID: schema.maybe(schema.string()),
}),
};

/**
* Used to validate GET requests for the ancestors of a process event.
*/
export const validateAncestry = {
params: schema.object({ id: schema.string() }),
query: schema.object({
ancestors: schema.number({ defaultValue: 0, min: 0, max: 10 }),
legacyEndpointID: schema.maybe(schema.string()),
}),
};

/**
* Used to validate GET requests for children of a specified process event.
*/
export const validateChildren = {
params: schema.object({ id: schema.string() }),
query: schema.object({
children: schema.number({ defaultValue: 10, min: 10, max: 100 }),
generations: schema.number({ defaultValue: 3, min: 0, max: 3 }),
afterChild: schema.maybe(schema.string()),
legacyEndpointID: schema.maybe(schema.string()),
}),
};
22 changes: 22 additions & 0 deletions x-pack/plugins/endpoint/common/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,28 @@ export class EndpointAppConstants {
static MAX_LONG_INT = '9223372036854775807'; // 2^63-1
}

export interface ResolverNodeStats {
totalEvents: number;
totalAlerts: number;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is totalEvents equal to the count of non-alert events + alert events?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this info? Isn't it expensive to get a count (esp. an exact count?)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is totalEvents equal to the count of non-alert events + alert events?

I think the info would useful when display totals in the UI.

Do we need this info? Isn't it expensive to get a count (esp. an exact count?)

I think this would be useful for displaying counts,

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❔ Does totalAlerts mean total related alerts? If so, can we add that to the interface name?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonathan-buttner if we showed the user some total, we would need to explain what the total is. What is this?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds like we're probably going to remove the stats stuff for now, but I think what this is trying to communicate is totalAlerts is the total number of alerts associated with a specific process node/entity_id aka how many alerts has this specific process generated.

totalEvents is how many related events exists for this process.

}

export interface ResolverNodePagination {
nextChild?: string | null;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❔ The ? here marks it as "possibly undefined" right? How does the meaning differ when it's null? Is null for when it is "last page" and undefined is for when it hasn't been paged yet?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ℹ️ Note to self to replace/merge some of the things in adjacencyMap with this so the meaning doesn't diverge

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❔ The ? here marks it as "possibly undefined" right? How does the meaning differ when it's null? Is null for when it is "last page" and undefined is for when it hasn't been paged yet?

Yeah I think null indicates that there are no more events of that type to retrieve. So you've reached the last page. If there are more pages, it'll be the entity_id that should be in the next request. I think we're using undefined here so it just doesn't show up in the pagination section. I could be wrong though. I don't think it is denoting something specific.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jonathan-buttner @kqualters-elastic if there is special meaning to these interface fields, can we document it here?

Also, can you clarify what "you've reached the last page" means? Data could still be coming in right? So unless we know that there will never be any more data, we will probably need to assume that there is more.

Let me know if I'm wrong or unclear. I'm writing everything on a tablet and it's tedious 😅

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO ask about this

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@oatkiller Yeah that's a fair point that we won't ever know for sure that there isn't more data because we could have been received some since they last time we searched, but it's probably still useful to indicate that we know for sure that there is more data available right?

For example if we only retrieve a portion of the children for a node, by indicating that we know there are more children we could display an arrow or plus sign or something in the UI indicating to the user that there is definitely more children to retrieve.

nextEvent?: string | null;
nextAncestor?: string | null;
nextAlert?: string | null;
}

export interface ResolverNode {
id: string;
children: ResolverNode[];
events: ResolverEvent[];
lifecycle: ResolverEvent[];
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how's this different than events? is this specifically process events with an entity_id that matches the id field on Node?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, lifecycle contains all the process events for a particular entityID including start, still_running, and end events. events are the related events for the node, i.e. file, registry, etc

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎗️

ancestors?: ResolverNode[];
pagination: ResolverNodePagination;
stats?: ResolverNodeStats;
}

export interface AlertResultList {
/**
* The alerts restricted by page size.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -33,5 +33,6 @@ export const AlertDetailResolver = styled(
width: 100%;
display: flex;
flex-grow: 1;
min-height: 500px;
/* gross demo hack */
min-height: calc(100vh - 505px);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❔ Did you check this on laptop-sized monitors? Seems like it would get squishy,

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that 505px is the static height of the alert detail content above it, not sure what choice we have really.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is an improvement, but the long term solution should involve flexbox or something

`;
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,11 @@ import { ResolverEvent } from '../../../../../common/types';

interface ServerReturnedResolverData {
readonly type: 'serverReturnedResolverData';
readonly payload: {
readonly data: {
readonly result: {
readonly search_results: readonly ResolverEvent[];
};
};
};
readonly payload: ResolverEvent[];
}

export type DataAction = ServerReturnedResolverData;
interface ServerFailedToReturnResolverData {
readonly type: 'serverFailedToReturnResolverData';
}

export type DataAction = ServerReturnedResolverData | ServerFailedToReturnResolverData;
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ import { Store, createStore } from 'redux';
import { DataAction } from './action';
import { dataReducer } from './reducer';
import { DataState } from '../../types';
import { LegacyEndpointEvent } from '../../../../../common/types';
import { LegacyEndpointEvent, ResolverEvent } from '../../../../../common/types';
import { graphableProcesses, processNodePositionsAndEdgeLineSegments } from './selectors';
import { mockProcessEvent } from '../../models/process_event_test_helpers';

Expand Down Expand Up @@ -113,13 +113,7 @@ describe('resolver graph layout', () => {
});
describe('when rendering no nodes', () => {
beforeEach(() => {
const payload = {
data: {
result: {
search_results: [],
},
},
};
const payload: ResolverEvent[] = [];
const action: DataAction = { type: 'serverReturnedResolverData', payload };
store.dispatch(action);
});
Expand All @@ -133,13 +127,7 @@ describe('resolver graph layout', () => {
});
describe('when rendering one node', () => {
beforeEach(() => {
const payload = {
data: {
result: {
search_results: [processA],
},
},
};
const payload = [processA];
const action: DataAction = { type: 'serverReturnedResolverData', payload };
store.dispatch(action);
});
Expand All @@ -153,13 +141,7 @@ describe('resolver graph layout', () => {
});
describe('when rendering two nodes, one being the parent of the other', () => {
beforeEach(() => {
const payload = {
data: {
result: {
search_results: [processA, processB],
},
},
};
const payload = [processA, processB];
const action: DataAction = { type: 'serverReturnedResolverData', payload };
store.dispatch(action);
});
Expand All @@ -173,23 +155,17 @@ describe('resolver graph layout', () => {
});
describe('when rendering two forks, and one fork has an extra long tine', () => {
beforeEach(() => {
const payload = {
data: {
result: {
search_results: [
processA,
processB,
processC,
processD,
processE,
processF,
processG,
processH,
processI,
],
},
},
};
const payload = [
processA,
processB,
processC,
processD,
processE,
processF,
processG,
processH,
processI,
];
const action: DataAction = { type: 'serverReturnedResolverData', payload };
store.dispatch(action);
});
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,25 +11,28 @@ function initialState(): DataState {
return {
results: [],
isLoading: false,
hasError: false,
};
}

export const dataReducer: Reducer<DataState, ResolverAction> = (state = initialState(), action) => {
if (action.type === 'serverReturnedResolverData') {
const {
data: {
result: { search_results },
},
} = action.payload;
return {
...state,
results: search_results,
results: action.payload,
isLoading: false,
hasError: false,
};
} else if (action.type === 'appRequestedResolverData') {
return {
...state,
isLoading: true,
hasError: false,
};
} else if (action.type === 'serverFailedToReturnResolverData') {
return {
...state,
hasError: true,
};
} else {
return state;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,10 @@ export function isLoading(state: DataState) {
return state.isLoading;
}

export function hasError(state: DataState) {
return state.hasError;
}

/**
* An isometric projection is a method for representing three dimensional objects in 2 dimensions.
* More information about isometric projections can be found here https://en.wikipedia.org/wiki/Isometric_projection.
Expand Down Expand Up @@ -293,7 +297,7 @@ function* levelOrderWithWidths(
metadata.firstChildWidth = width;
} else {
const firstChildWidth = widths.get(siblings[0]);
const lastChildWidth = widths.get(siblings[0]);
const lastChildWidth = widths.get(siblings[siblings.length - 1]);
if (firstChildWidth === undefined || lastChildWidth === undefined) {
/**
* All widths have been precalcluated, so this will not happen.
Expand Down
Loading