Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution][Detections][Threshold Rules] Threshold multiple aggregations with cardinality #90826

Merged
merged 50 commits into from
Feb 18, 2021
Merged
Changes from 1 commit
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
a4139ce
Remove unnecessary spreads
madirey Jan 14, 2021
b73b34a
Layout, round 1
madirey Jan 14, 2021
026a6d7
Merge branch 'master' of github.com:elastic/kibana into threshold-mul…
madirey Jan 25, 2021
8c5c889
Merge branch 'master' of github.com:elastic/kibana into threshold-mul…
madirey Jan 27, 2021
ed5f9b0
Revert "Layout, round 1"
madirey Jan 28, 2021
b2f31ba
Make threshold field an array
madirey Jan 28, 2021
0842e84
Add cardinality fields
madirey Jan 28, 2021
02c3b2c
Fix validation schema
madirey Feb 1, 2021
48f6545
Query for multi-aggs
madirey Feb 1, 2021
e435241
Merge branch 'master' of github.com:elastic/kibana into threshold-mul…
madirey Feb 1, 2021
70e4f50
Finish multi-agg aggregation
madirey Feb 1, 2021
cf7ef93
Translate to multi-agg buckets
madirey Feb 7, 2021
1ebfed0
Fix existing tests and add new test skeletons
madirey Feb 7, 2021
0786757
merge master, fix conflicts
madirey Feb 7, 2021
12a98bb
clean up
madirey Feb 9, 2021
6cba63a
Fix types
marshallmain Feb 11, 2021
78e77bc
Fix threshold_result data structure
madirey Feb 11, 2021
465c5a4
previous signals filter
madirey Feb 11, 2021
18d5363
Fix previous signal detection
madirey Feb 12, 2021
41a5ddb
Finish previous signal parsing
madirey Feb 12, 2021
19ed253
tying up loose ends
madirey Feb 14, 2021
733347c
merge master, fix conflicts
madirey Feb 15, 2021
c7eea31
Merge branch 'master' of github.com:elastic/kibana into threshold-mul…
madirey Feb 15, 2021
319e9db
Fix timeline view for multi-agg threshold signals
madirey Feb 15, 2021
e2a7d40
Fix build_bulk_body tests
madirey Feb 15, 2021
c6abdf5
test fixes
madirey Feb 16, 2021
741c75e
Add test for threshold bucket filters
madirey Feb 16, 2021
b277b04
Address comments
madirey Feb 16, 2021
8d1e922
Fixing schema errors
madirey Feb 16, 2021
6b8c8ed
Remove unnecessary comment
madirey Feb 16, 2021
a8dc733
Fix tests
madirey Feb 16, 2021
6fd0836
Fix types
madirey Feb 16, 2021
900ead0
Merge branch 'master' of github.com:elastic/kibana into threshold-mul…
madirey Feb 16, 2021
37956ca
linting
madirey Feb 17, 2021
7956953
linting
madirey Feb 17, 2021
af5ed84
Fixes
madirey Feb 17, 2021
ee103c1
Handle pre-7.12 threshold format in timeline view
madirey Feb 17, 2021
2094d58
missing null check
madirey Feb 17, 2021
bed1faf
adding in follow-up pr
madirey Feb 17, 2021
e979d12
Handle pre-7.12 filters
madirey Feb 17, 2021
9dab01e
Merge branch 'master' of github.com:elastic/kibana into threshold-mul…
madirey Feb 17, 2021
3edc7f2
unnecessary change
madirey Feb 17, 2021
13821bf
Revert "unnecessary change"
madirey Feb 17, 2021
f88cf66
linting
madirey Feb 17, 2021
3e09b24
Fix rule schemas
madirey Feb 17, 2021
6eafa8d
Merge branch 'master' of github.com:elastic/kibana into threshold-mul…
madirey Feb 17, 2021
db6dfa5
Fix tests
madirey Feb 17, 2021
9ba09b6
Merge branch 'master' of github.com:elastic/kibana into threshold-mul…
madirey Feb 18, 2021
b6fd98b
merge master, fix conflicts
madirey Feb 18, 2021
5c503fc
more fixing conflicts
madirey Feb 18, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix types
marshallmain committed Feb 11, 2021

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 6cba63a357f2b8309345feba7506a02c6d8fe13a
Original file line number Diff line number Diff line change
@@ -36,7 +36,14 @@ export interface MatrixHistogramRequestOptions extends RequestBasicOptions {
timerange: TimerangeInput;
histogramType: MatrixHistogramType;
stackByField: string;
threshold?: { field: string | undefined; value: number } | undefined;
threshold?:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like a few changes similar to this one were made in this PR, would it be worth pulling this out into its own type shared between the updated code? not a blocker, but might be a nice follow up.

| {
field: string | string[] | undefined;
value: number;
cardinality_field?: string;
cardinality_value?: number;
}
| undefined;
inspect?: Maybe<Inspect>;
isPtrIncluded?: boolean;
}
Original file line number Diff line number Diff line change
@@ -288,7 +288,12 @@ describe('PreviewQuery', () => {
idAria="queryPreview"
query={{ query: { query: 'file where true', language: 'kuery' }, filters: [] }}
index={['foo-*']}
threshold={{ field: 'agent.hostname', value: 200 }}
threshold={{
field: 'agent.hostname',
value: 200,
cardinality_field: 'user.name',
cardinality_value: 2,
}}
isDisabled={false}
/>
</TestProviders>
@@ -330,7 +335,12 @@ describe('PreviewQuery', () => {
idAria="queryPreview"
query={{ query: { query: 'file where true', language: 'kuery' }, filters: [] }}
index={['foo-*']}
threshold={{ field: 'agent.hostname', value: 200 }}
threshold={{
field: 'agent.hostname',
value: 200,
cardinality_field: 'user.name',
cardinality_value: 2,
}}
isDisabled={false}
/>
</TestProviders>
@@ -369,7 +379,12 @@ describe('PreviewQuery', () => {
idAria="queryPreview"
query={{ query: { query: 'file where true', language: 'kuery' }, filters: [] }}
index={['foo-*']}
threshold={{ field: undefined, value: 200 }}
threshold={{
field: undefined,
value: 200,
cardinality_field: 'user.name',
cardinality_value: 2,
}}
isDisabled={false}
/>
</TestProviders>
@@ -396,7 +411,12 @@ describe('PreviewQuery', () => {
idAria="queryPreview"
query={{ query: { query: 'file where true', language: 'kuery' }, filters: [] }}
index={['foo-*']}
threshold={{ field: ' ', value: 200 }}
threshold={{
field: ' ',
value: 200,
cardinality_field: 'user.name',
cardinality_value: 2,
}}
isDisabled={false}
/>
</TestProviders>
Original file line number Diff line number Diff line change
@@ -334,7 +334,12 @@ describe('queryPreviewReducer', () => {
test('should set thresholdFieldExists to true if threshold field is defined and not empty string', () => {
const update = reducer(initialState, {
type: 'setThresholdQueryVals',
threshold: { field: 'agent.hostname', value: 200 },
threshold: {
field: 'agent.hostname',
value: 200,
cardinality_field: 'user.name',
cardinality_value: 2,
},
ruleType: 'threshold',
});

@@ -347,7 +352,12 @@ describe('queryPreviewReducer', () => {
test('should set thresholdFieldExists to false if threshold field is not defined', () => {
const update = reducer(initialState, {
type: 'setThresholdQueryVals',
threshold: { field: undefined, value: 200 },
threshold: {
field: undefined,
value: 200,
cardinality_field: 'user.name',
cardinality_value: 2,
},
ruleType: 'threshold',
});

@@ -360,7 +370,12 @@ describe('queryPreviewReducer', () => {
test('should set thresholdFieldExists to false if threshold field is empty string', () => {
const update = reducer(initialState, {
type: 'setThresholdQueryVals',
threshold: { field: ' ', value: 200 },
threshold: {
field: ' ',
value: 200,
cardinality_field: 'user.name',
cardinality_value: 2,
},
ruleType: 'threshold',
});

@@ -373,7 +388,12 @@ describe('queryPreviewReducer', () => {
test('should set showNonEqlHistogram to false if ruleType is eql', () => {
const update = reducer(initialState, {
type: 'setThresholdQueryVals',
threshold: { field: 'agent.hostname', value: 200 },
threshold: {
field: 'agent.hostname',
value: 200,
cardinality_field: 'user.name',
cardinality_value: 2,
},
ruleType: 'eql',
});

@@ -385,7 +405,12 @@ describe('queryPreviewReducer', () => {
test('should set showNonEqlHistogram to true if ruleType is query', () => {
const update = reducer(initialState, {
type: 'setThresholdQueryVals',
threshold: { field: 'agent.hostname', value: 200 },
threshold: {
field: 'agent.hostname',
value: 200,
cardinality_field: 'user.name',
cardinality_value: 2,
},
ruleType: 'query',
});

@@ -397,7 +422,12 @@ describe('queryPreviewReducer', () => {
test('should set showNonEqlHistogram to true if ruleType is saved_query', () => {
const update = reducer(initialState, {
type: 'setThresholdQueryVals',
threshold: { field: 'agent.hostname', value: 200 },
threshold: {
field: 'agent.hostname',
value: 200,
cardinality_field: 'user.name',
cardinality_value: 2,
},
ruleType: 'saved_query',
});

Original file line number Diff line number Diff line change
@@ -131,7 +131,9 @@ export const queryPreviewReducer = () => (state: State, action: Action): State =
const thresholdField =
action.threshold != null &&
action.threshold.field != null &&
action.threshold.field.trim() !== '';
((typeof action.threshold.field === 'string' && action.threshold.field.trim() !== '') ||
(Array.isArray(action.threshold.field) &&
action.threshold.field.every((field) => field.trim() !== '')));
const showNonEqlHist =
action.ruleType === 'query' ||
action.ruleType === 'saved_query' ||
Original file line number Diff line number Diff line change
@@ -83,7 +83,7 @@ const stepDefineDefaultValue: DefineStepRule = {
field: [],
value: '200',
cardinality_field: [],
cardinality_value: '2',
cardinality_value: 2,
},
timeline: {
id: null,
Original file line number Diff line number Diff line change
@@ -20,7 +20,7 @@ export interface FieldValueThreshold {
field: string[];
value: string;
cardinality_field: string[];
cardinality_value: string;
cardinality_value: number;
}

interface ThresholdInputProps {
Original file line number Diff line number Diff line change
@@ -192,6 +192,8 @@ export const mockDefineStepRule = (): DefineStepRule => ({
threshold: {
field: [''],
value: '100',
cardinality_field: [''],
cardinality_value: 2,
},
});

Original file line number Diff line number Diff line change
@@ -221,8 +221,8 @@ export const formatDefineStepData = (defineStepData: DefineStepRule): DefineStep
threshold: {
field: ruleFields.threshold?.field ?? [],
value: parseInt(ruleFields.threshold?.value, 10) ?? 0,
cardinality_field: ruleFields.threshold?.cardinality_field[0] ?? '',
cardinality_value: parseInt(ruleFields.threshold?.cardinality_value, 10) ?? 0,
cardinality_field: ruleFields.threshold.cardinality_field[0] ?? '',
cardinality_value: ruleFields.threshold.cardinality_value,
},
}),
}
Original file line number Diff line number Diff line change
@@ -99,8 +99,14 @@ export const getDefineStepsData = (rule: Rule): DefineStepRule => ({
title: rule.timeline_title ?? null,
},
threshold: {
field: rule.threshold?.field ? [rule.threshold.field] : [],
field: rule.threshold?.field
? Array.isArray(rule.threshold.field)
? rule.threshold.field
: [rule.threshold.field]
: [],
value: `${rule.threshold?.value || 100}`,
cardinality_field: rule.threshold?.cardinality_field ? [rule.threshold?.cardinality_field] : [],
cardinality_value: rule.threshold?.cardinality_value ?? 2,
},
});

Original file line number Diff line number Diff line change
@@ -18,13 +18,13 @@ import {
AlertInstanceState,
AlertServices,
} from '../../../../../alerts/server';
import { RuleAlertAction } from '../../../../common/detection_engine/types';
import { BaseHit, RuleAlertAction } from '../../../../common/detection_engine/types';
import { RuleTypeParams, RefreshTypes } from '../types';
import { singleBulkCreate, SingleBulkCreateResponse } from './single_bulk_create';
import { calculateThresholdSignalUuid } from './utils';
import { BuildRuleMessage } from './rule_messages';
import { TermAggregationBucket } from '../../types';
import { MultiAggBucket, SignalSearchResponse } from './types';
import { MultiAggBucket, SignalSearchResponse, SignalSource } from './types';

interface BulkCreateThresholdSignalsParams {
actions: RuleAlertAction[];
@@ -60,7 +60,7 @@ const getTransformedHits = (
ruleId: string,
filter: unknown,
timestampOverride: TimestampOverrideOrUndefined
): Array<BaseHit<SignalSource>> => {
) => {
if (isEmpty(threshold.field)) {
const totalResults =
typeof results.hits.total === 'number' ? results.hits.total : results.hits.total.value;
@@ -74,9 +74,17 @@ const getTransformedHits = (
logger.warn(`No hits returned, but totalResults >= threshold.value (${threshold.value})`);
return [];
}
const timestampArray = get(timestampOverride ?? '@timestamp', hit.fields);
if (timestampArray == null) {
return [];
}
const timestamp = timestampArray[0];
if (typeof timestamp !== 'string') {
return [];
}

const source = {
'@timestamp': get(timestampOverride ?? '@timestamp', hit.fields),
'@timestamp': timestamp,
threshold_result: {
count: totalResults,
value: ruleId,
@@ -126,23 +134,32 @@ const getTransformedHits = (
}, []);
};

return getCombinations(results.aggregations.threshold_0.buckets, 0)
.map((bucket) => {
return getCombinations(results.aggregations.threshold_0.buckets, 0).reduce(
(acc: Array<BaseHit<SignalSource>>, bucket) => {
const hit = bucket.topThresholdHits?.hits.hits[0];
if (hit == null) {
return null;
return acc;
}

const timestampArray = get(timestampOverride ?? '@timestamp', hit.fields);
if (timestampArray == null) {
return [];
}
const timestamp = timestampArray[0];
if (typeof timestamp !== 'string') {
return [];
}

const source = {
'@timestamp': get(timestampOverride ?? '@timestamp', hit._source),
'@timestamp': timestamp,
threshold_result: {
count: bucket.docCount,
value: bucket.terms,
cardinality_count: bucket.cardinalityCount,
},
};

return {
acc.push({
_index: inputIndex,
_id: calculateThresholdSignalUuid(
ruleId,
@@ -151,9 +168,11 @@ const getTransformedHits = (
bucket.terms.join(',')
),
_source: source,
};
})
.filter((bucket) => bucket != null);
});
return acc;
},
[]
);
};

export const transformThresholdResultsToEcs = (
Original file line number Diff line number Diff line change
@@ -51,7 +51,7 @@ export interface SignalsStatusParams {

export interface ThresholdResult {
count: number;
value: string;
value: string | string[];
}

export interface SignalSource {
@@ -280,7 +280,7 @@ export interface ThresholdAggregationBucket extends TermAggregationBucket {

export interface MultiAggBucket {
terms: string[];
docCount?: number | undefined;
docCount: number;
topThresholdHits?:
| {
hits: {
Original file line number Diff line number Diff line change
@@ -1445,13 +1445,13 @@ describe('utils', () => {
describe('calculateThresholdSignalUuid', () => {
it('should generate a uuid without key', () => {
const startedAt = new Date('2020-12-17T16:27:00Z');
const signalUuid = calculateThresholdSignalUuid('abcd', startedAt, 'agent.name');
const signalUuid = calculateThresholdSignalUuid('abcd', startedAt, ['agent.name']);
expect(signalUuid).toEqual('a4832768-a379-583a-b1a2-e2ce2ad9e6e9');
});

it('should generate a uuid with key', () => {
const startedAt = new Date('2019-11-18T13:32:00Z');
const signalUuid = calculateThresholdSignalUuid('abcd', startedAt, 'host.ip', '1.2.3.4');
const signalUuid = calculateThresholdSignalUuid('abcd', startedAt, ['host.ip'], '1.2.3.4');
expect(signalUuid).toEqual('ee8870dc-45ff-5e6c-a2f9-80886651ce03');
});
});
6 changes: 3 additions & 3 deletions x-pack/plugins/security_solution/server/lib/types.ts
Original file line number Diff line number Diff line change
@@ -75,8 +75,8 @@ export interface SearchHits<T> {
max_score: number;
hits: Array<
BaseHit<T> & {
_type: string;
_score: number;
_type?: string;
_score?: number;
_version?: number;
_explanation?: Explanation;
// eslint-disable-next-line @typescript-eslint/no-explicit-any
@@ -106,7 +106,7 @@ export type SearchHit = SearchResponse<object>['hits']['hits'][0];

export interface TermAggregationBucket {
key: string;
doc_count: number | undefined;
doc_count: number;
top_threshold_hits?: {
hits: {
hits: SearchHit[];