[Security Solution][Detections] Reading last 5 failures from Event Lo…

…g v1 - raw implementation (#115574) (#116947) **Ticket:** #106469, #101013 ## Summary TL;DR: New internal endpoint for reading data from Event Log (raw version), legacy status SO under the hood. With this PR we now read the Failure History (last 5 failures) on the Rule Details page from Event Log. We continue getting the Current Status from the legacy `siem-detection-engine-rule-status` saved objects. Rule Management page also gets data from the legacy saved objects. - [x] Deprecate existing methods for reading data in `IRuleExecutionLogClient`: `.find()` and `.findBulk()` - [x] Introduce new methods for reading data in IRuleExecutionLogClient: - for reading last N execution events for 1 rule from event log - for reading current status and metrics for 1 rule from legacy status SOs - for reading current statuses and metrics for N rules from legacy status SOs - [x] New methods should return data in the legacy status SO format. - [x] Update all the existing endpoints that depend on `IRuleExecutionLogClient` to use the new methods. - [x] Implement a new internal endpoint for fetching current status of the rule execution and execution events from Event Log for a given rule. - [x] The API of the new endpoint should be the same as `rules/_find_statuses` to minimise changes in the app. - [x] Use the new endpoint on the Rule Details page. ## Near-term plan for technical implementation of the Rule Execution Log (#101013) **Stage 1. Reading last 5 failures from Event Log v1 - raw implementation** - ✔️ done in this PR TL;DR: New internal endpoint for reading data from Event Log (raw version), legacy status SO under the hood. - Deprecate existing methods for reading data in `IRuleExecutionLogClient`: `.find()` and `.findBulk()` - Introduce new methods for reading data in IRuleExecutionLogClient: - for reading last N execution events for 1 rule from event log - for reading current status and metrics for 1 rule from legacy status SOs - for reading current statuses and metrics for N rules from legacy status SOs - New methods should return data in the legacy status SO format. - Update all the existing endpoints that depend on `IRuleExecutionLogClient` to use the new methods. - Implement a new internal endpoint for fetching current status of the rule execution and execution events from Event Log for a given rule. - The API of the new endpoint should be the same as `rules/_find_statuses` to minimise changes in the app. - Use the new endpoint on the Rule Details page. **Stage 2: Reading last 5 failures from Event Log v2 - clean implementation** TL;DR: Clean HTTP API, legacy Rule Status SO under the hood. 🚨🚨🚨 Possible breaking changes in Detections API 🚨🚨🚨 - Design a new data model for the Current Rule Execution Info (the TO-BE new SO type and later the TO-BE data in the rule object itself). - Design a new data model for the Rule Execution Event (read model to be used on the Rule Details page) - Think over changes in `IRuleExecutionLogClient` to support the new data model. - Think over changes in all the endpoints that return any data related to rule monitoring (statuses, metrics, etc). Make sure to check our docs to identify what's documented there regarding rule monitoring. - Update `IRuleExecutionLogClient` to return data in the new format. - Update all the endpoints (including the raw new one) to return data in the new format. - Update Rule Details page to consume data in the new format. - Update Rule Management page to consume data in the new format. **Stage 3: Reading last 5 failures from Event Log v3 - new SO** TL;DR: Clean HTTP API, new Rule Execution Info SO under the hood. - Implement a new SO type for storing the current rule execution info. Relation type: 1 rule - 1 current execution info. - Swap the legacy SO with the new SO in the implementation of `IRuleExecutionLogClient`. **Stage 4: Cleanup and misc** - Revisit the problem of deterministic ordering ([comment](#115574 (comment))) - Remove rule execution log's glue code: adapters, feature switch. - Remove the legacy rule status SO. - Mark the legacy rule status SO as deleted in Kibana Core. - Encapsulate the current space id in the instance of IRuleExecutionLogClient. Remove it from parameters of its methods. - Introduce a Rule Execution Logger scoped to a rule instance. For use in rule executors. - Add test coverage. ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios Co-authored-by: Georgii Gorbachev <[email protected]>
elastic · Nov 1, 2021 · f241f62 · f241f62
1 parent fe35e23
commit f241f62
Show file tree

Hide file tree

Showing 43 changed files with 696 additions and 390 deletions.
diff --git a/x-pack/plugins/security_solution/common/constants.ts b/x-pack/plugins/security_solution/common/constants.ts
@@ -250,6 +250,13 @@ export const DETECTION_ENGINE_RULES_PREVIEW = `${DETECTION_ENGINE_RULES_URL}/pre
 export const DETECTION_ENGINE_RULES_PREVIEW_INDEX_URL =
   `${DETECTION_ENGINE_RULES_PREVIEW}/index` as const;
 
+/**
+ * Internal detection engine routes
+ */
+export const INTERNAL_DETECTION_ENGINE_URL = '/internal/detection_engine' as const;
+export const INTERNAL_DETECTION_ENGINE_RULE_STATUS_URL =
+  `${INTERNAL_DETECTION_ENGINE_URL}/rules/_find_status` as const;
+
 export const TIMELINE_RESOLVE_URL = '/api/timeline/resolve' as const;
 export const TIMELINE_URL = '/api/timeline' as const;
 export const TIMELINES_URL = '/api/timelines' as const;

diff --git a/...ns/security_solution/common/detection_engine/schemas/request/find_rule_statuses_schema.ts b/...ns/security_solution/common/detection_engine/schemas/request/find_rule_statuses_schema.ts
@@ -16,3 +16,13 @@ export const findRulesStatusesSchema = t.exact(
 export type FindRulesStatusesSchema = t.TypeOf<typeof findRulesStatusesSchema>;
 
 export type FindRulesStatusesSchemaDecoded = FindRulesStatusesSchema;
+
+export const findRuleStatusSchema = t.exact(
+  t.type({
+    ruleId: t.string,
+  })
+);
+
+export type FindRuleStatusSchema = t.TypeOf<typeof findRuleStatusSchema>;
+
+export type FindRuleStatusSchemaDecoded = FindRuleStatusSchema;
diff --git a/...plugins/security_solution/public/detections/containers/detection_engine/rules/api.test.ts b/...plugins/security_solution/public/detections/containers/detection_engine/rules/api.test.ts
@@ -669,8 +669,8 @@ describe('Detections Rules API', () => {
 
     test('check parameter url, query', async () => {
       await getRuleStatusById({ id: 'mySuperRuleId', signal: abortCtrl.signal });
-      expect(fetchMock).toHaveBeenCalledWith('/api/detection_engine/rules/_find_statuses', {
-        body: '{"ids":["mySuperRuleId"]}',
+      expect(fetchMock).toHaveBeenCalledWith('/internal/detection_engine/rules/_find_status', {
+        body: '{"ruleId":"mySuperRuleId"}',
         method: 'POST',
         signal: abortCtrl.signal,
       });

diff --git a/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.ts b/x-pack/plugins/security_solution/public/detections/containers/detection_engine/rules/api.ts
@@ -19,6 +19,7 @@ import {
   DETECTION_ENGINE_TAGS_URL,
   DETECTION_ENGINE_RULES_BULK_ACTION,
   DETECTION_ENGINE_RULES_PREVIEW,
+  INTERNAL_DETECTION_ENGINE_RULE_STATUS_URL,
 } from '../../../../../common/constants';
 import {
   UpdateRulesProps,
@@ -372,9 +373,9 @@ export const getRuleStatusById = async ({
   id: string;
   signal: AbortSignal;
 }): Promise<RuleStatusResponse> =>
-  KibanaServices.get().http.fetch<RuleStatusResponse>(DETECTION_ENGINE_RULES_STATUS_URL, {
+  KibanaServices.get().http.fetch<RuleStatusResponse>(INTERNAL_DETECTION_ENGINE_RULE_STATUS_URL, {
     method: 'POST',
-    body: JSON.stringify({ ids: [id] }),
+    body: JSON.stringify({ ruleId: id }),
     signal,
   });
 

diff --git a/...ugins/security_solution/server/lib/detection_engine/routes/__mocks__/request_responses.ts b/...ugins/security_solution/server/lib/detection_engine/routes/__mocks__/request_responses.ts
@@ -9,7 +9,7 @@ import type * as estypes from '@elastic/elasticsearch/lib/api/typesWithBodyKey';
 import { ALERT_WORKFLOW_STATUS } from '@kbn/rule-data-utils';
 import { ruleTypeMappings } from '@kbn/securitysolution-rules';
 
-import { SavedObjectsFindResponse, SavedObjectsFindResult } from 'kibana/server';
+import { SavedObjectsFindResponse } from 'src/core/server';
 
 import { ActionResult } from '../../../../../../actions/server';
 import {
@@ -23,6 +23,7 @@ import {
   DETECTION_ENGINE_SIGNALS_FINALIZE_MIGRATION_URL,
   DETECTION_ENGINE_SIGNALS_MIGRATION_STATUS_URL,
   DETECTION_ENGINE_RULES_BULK_ACTION,
+  INTERNAL_DETECTION_ENGINE_RULE_STATUS_URL,
 } from '../../../../../common/constants';
 import {
   RuleAlertType,
@@ -42,7 +43,7 @@ import { SanitizedAlert, ResolvedSanitizedRule } from '../../../../../../alertin
 import { getQueryRuleParams } from '../../schemas/rule_schemas.mock';
 import { getPerformBulkActionSchemaMock } from '../../../../../common/detection_engine/schemas/request/perform_bulk_action_schema.mock';
 import { RuleExecutionStatus } from '../../../../../common/detection_engine/schemas/common/schemas';
-import { FindBulkExecutionLogResponse } from '../../rule_execution_log/types';
+import { GetCurrentStatusBulkResult } from '../../rule_execution_log/types';
 // eslint-disable-next-line no-restricted-imports
 import type { LegacyRuleNotificationAlertType } from '../../notifications/legacy_types';
 
@@ -232,6 +233,13 @@ export const ruleStatusRequest = () =>
     body: { ids: ['04128c15-0d1b-4716-a4c5-46997ac7f3bd'] },
   });
 
+export const internalRuleStatusRequest = () =>
+  requestMock.create({
+    method: 'post',
+    path: INTERNAL_DETECTION_ENGINE_RULE_STATUS_URL,
+    body: { ruleId: '04128c15-0d1b-4716-a4c5-46997ac7f3bd' },
+  });
+
 export const getImportRulesRequest = (hapiStream?: HapiReadableStream) =>
   requestMock.create({
     method: 'post',
@@ -475,94 +483,64 @@ export const getEmptySavedObjectsResponse =
     saved_objects: [],
   });
 
-export const getRuleExecutionStatuses = (): Array<
-  SavedObjectsFindResult<IRuleStatusSOAttributes>
-> => [
-  {
-    type: 'my-type',
-    id: 'e0b86950-4e9f-11ea-bdbd-07b56aa159b3',
-    attributes: {
-      statusDate: '2020-02-18T15:26:49.783Z',
-      status: RuleExecutionStatus.succeeded,
-      lastFailureAt: undefined,
-      lastSuccessAt: '2020-02-18T15:26:49.783Z',
-      lastFailureMessage: undefined,
-      lastSuccessMessage: 'succeeded',
-      lastLookBackDate: new Date('2020-02-18T15:14:58.806Z').toISOString(),
-      gap: '500.32',
-      searchAfterTimeDurations: ['200.00'],
-      bulkCreateTimeDurations: ['800.43'],
-    },
-    score: 1,
-    references: [
-      {
-        id: '04128c15-0d1b-4716-a4c5-46997ac7f3bc',
-        type: 'alert',
-        name: 'alert_0',
-      },
-    ],
-    updated_at: '2020-02-18T15:26:51.333Z',
-    version: 'WzQ2LDFd',
-  },
-  {
-    type: 'my-type',
-    id: '91246bd0-5261-11ea-9650-33b954270f67',
-    attributes: {
-      statusDate: '2020-02-18T15:15:58.806Z',
-      status: RuleExecutionStatus.failed,
-      lastFailureAt: '2020-02-18T15:15:58.806Z',
-      lastSuccessAt: '2020-02-13T20:31:59.855Z',
-      lastFailureMessage:
-        'Signal rule name: "Query with a rule id Number 1", id: "1ea5a820-4da1-4e82-92a1-2b43a7bece08", rule_id: "query-rule-id-1" has a time gap of 5 days (412682928ms), and could be missing signals within that time. Consider increasing your look behind time or adding more Kibana instances.',
-      lastSuccessMessage: 'succeeded',
-      lastLookBackDate: new Date('2020-02-18T15:14:58.806Z').toISOString(),
-      gap: '500.32',
-      searchAfterTimeDurations: ['200.00'],
-      bulkCreateTimeDurations: ['800.43'],
-    },
-    score: 1,
-    references: [
-      {
-        id: '1ea5a820-4da1-4e82-92a1-2b43a7bece08',
-        type: 'alert',
-        name: 'alert_0',
-      },
-    ],
-    updated_at: '2020-02-18T15:15:58.860Z',
-    version: 'WzMyLDFd',
-  },
+export const getRuleExecutionStatusSucceeded = (): IRuleStatusSOAttributes => ({
+  statusDate: '2020-02-18T15:26:49.783Z',
+  status: RuleExecutionStatus.succeeded,
+  lastFailureAt: undefined,
+  lastSuccessAt: '2020-02-18T15:26:49.783Z',
+  lastFailureMessage: undefined,
+  lastSuccessMessage: 'succeeded',
+  lastLookBackDate: new Date('2020-02-18T15:14:58.806Z').toISOString(),
+  gap: '500.32',
+  searchAfterTimeDurations: ['200.00'],
+  bulkCreateTimeDurations: ['800.43'],
+});
+
+export const getRuleExecutionStatusFailed = (): IRuleStatusSOAttributes => ({
+  statusDate: '2020-02-18T15:15:58.806Z',
+  status: RuleExecutionStatus.failed,
+  lastFailureAt: '2020-02-18T15:15:58.806Z',
+  lastSuccessAt: '2020-02-13T20:31:59.855Z',
+  lastFailureMessage:
+    'Signal rule name: "Query with a rule id Number 1", id: "1ea5a820-4da1-4e82-92a1-2b43a7bece08", rule_id: "query-rule-id-1" has a time gap of 5 days (412682928ms), and could be missing signals within that time. Consider increasing your look behind time or adding more Kibana instances.',
+  lastSuccessMessage: 'succeeded',
+  lastLookBackDate: new Date('2020-02-18T15:14:58.806Z').toISOString(),
+  gap: '500.32',
+  searchAfterTimeDurations: ['200.00'],
+  bulkCreateTimeDurations: ['800.43'],
+});
+
+export const getRuleExecutionStatuses = (): IRuleStatusSOAttributes[] => [
+  getRuleExecutionStatusSucceeded(),
+  getRuleExecutionStatusFailed(),
 ];
 
-export const getFindBulkResultStatus = (): FindBulkExecutionLogResponse => ({
-  '04128c15-0d1b-4716-a4c5-46997ac7f3bd': [
-    {
-      statusDate: '2020-02-18T15:26:49.783Z',
-      status: RuleExecutionStatus.succeeded,
-      lastFailureAt: undefined,
-      lastSuccessAt: '2020-02-18T15:26:49.783Z',
-      lastFailureMessage: undefined,
-      lastSuccessMessage: 'succeeded',
-      lastLookBackDate: new Date('2020-02-18T15:14:58.806Z').toISOString(),
-      gap: '500.32',
-      searchAfterTimeDurations: ['200.00'],
-      bulkCreateTimeDurations: ['800.43'],
-    },
-  ],
-  '1ea5a820-4da1-4e82-92a1-2b43a7bece08': [
-    {
-      statusDate: '2020-02-18T15:15:58.806Z',
-      status: RuleExecutionStatus.failed,
-      lastFailureAt: '2020-02-18T15:15:58.806Z',
-      lastSuccessAt: '2020-02-13T20:31:59.855Z',
-      lastFailureMessage:
-        'Signal rule name: "Query with a rule id Number 1", id: "1ea5a820-4da1-4e82-92a1-2b43a7bece08", rule_id: "query-rule-id-1" has a time gap of 5 days (412682928ms), and could be missing signals within that time. Consider increasing your look behind time or adding more Kibana instances.',
-      lastSuccessMessage: 'succeeded',
-      lastLookBackDate: new Date('2020-02-18T15:14:58.806Z').toISOString(),
-      gap: '500.32',
-      searchAfterTimeDurations: ['200.00'],
-      bulkCreateTimeDurations: ['800.43'],
-    },
-  ],
+export const getFindBulkResultStatus = (): GetCurrentStatusBulkResult => ({
+  '04128c15-0d1b-4716-a4c5-46997ac7f3bd': {
+    statusDate: '2020-02-18T15:26:49.783Z',
+    status: RuleExecutionStatus.succeeded,
+    lastFailureAt: undefined,
+    lastSuccessAt: '2020-02-18T15:26:49.783Z',
+    lastFailureMessage: undefined,
+    lastSuccessMessage: 'succeeded',
+    lastLookBackDate: new Date('2020-02-18T15:14:58.806Z').toISOString(),
+    gap: '500.32',
+    searchAfterTimeDurations: ['200.00'],
+    bulkCreateTimeDurations: ['800.43'],
+  },
+  '1ea5a820-4da1-4e82-92a1-2b43a7bece08': {
+    statusDate: '2020-02-18T15:15:58.806Z',
+    status: RuleExecutionStatus.failed,
+    lastFailureAt: '2020-02-18T15:15:58.806Z',
+    lastSuccessAt: '2020-02-13T20:31:59.855Z',
+    lastFailureMessage:
+      'Signal rule name: "Query with a rule id Number 1", id: "1ea5a820-4da1-4e82-92a1-2b43a7bece08", rule_id: "query-rule-id-1" has a time gap of 5 days (412682928ms), and could be missing signals within that time. Consider increasing your look behind time or adding more Kibana instances.',
+    lastSuccessMessage: 'succeeded',
+    lastLookBackDate: new Date('2020-02-18T15:14:58.806Z').toISOString(),
+    gap: '500.32',
+    searchAfterTimeDurations: ['200.00'],
+    bulkCreateTimeDurations: ['800.43'],
+  },
 });
 
 export const getBasicEmptySearchResponse = (): estypes.SearchResponse<unknown> => ({

diff --git a/...ins/security_solution/server/lib/detection_engine/routes/rules/create_rules_route.test.ts b/...ins/security_solution/server/lib/detection_engine/routes/rules/create_rules_route.test.ts
@@ -10,7 +10,7 @@ import {
   getEmptyFindResult,
   getAlertMock,
   getCreateRequest,
-  getRuleExecutionStatuses,
+  getRuleExecutionStatusSucceeded,
   getFindResultWithSingleHit,
   createMlRuleRequest,
   getBasicEmptySearchResponse,
@@ -43,7 +43,9 @@ describe.each([
     clients.rulesClient.create.mockResolvedValue(
       getAlertMock(isRuleRegistryEnabled, getQueryRuleParams())
     ); // creation succeeds
-    clients.ruleExecutionLogClient.find.mockResolvedValue(getRuleExecutionStatuses()); // needed to transform: ;
+    clients.ruleExecutionLogClient.getCurrentStatus.mockResolvedValue(
+      getRuleExecutionStatusSucceeded()
+    );
 
     context.core.elasticsearch.client.asCurrentUser.search.mockResolvedValue(
       elasticsearchClientMock.createSuccessTransportRequestPromise(getBasicEmptySearchResponse())

diff --git a/.../plugins/security_solution/server/lib/detection_engine/routes/rules/create_rules_route.ts b/.../plugins/security_solution/server/lib/detection_engine/routes/rules/create_rules_route.ts
@@ -106,14 +106,13 @@ export const createRulesRoute = (
           await rulesClient.muteAll({ id: createdRule.id });
         }
 
-        const ruleStatuses = await context.securitySolution.getExecutionLogClient().find({
-          logsCount: 1,
+        const ruleStatus = await context.securitySolution.getExecutionLogClient().getCurrentStatus({
           ruleId: createdRule.id,
           spaceId: context.securitySolution.getSpaceId(),
         });
         const [validated, errors] = newTransformValidate(
           createdRule,
-          ruleStatuses[0],
+          ruleStatus,
           isRuleRegistryEnabled
         );
         if (errors != null) {

diff --git a/...ins/security_solution/server/lib/detection_engine/routes/rules/delete_rules_bulk_route.ts b/...ins/security_solution/server/lib/detection_engine/routes/rules/delete_rules_bulk_route.ts
@@ -80,21 +80,19 @@ export const deleteRulesBulkRoute = (
             return getIdBulkError({ id, ruleId });
           }
 
-          const ruleStatuses = await ruleStatusClient.find({
-            logsCount: 6,
+          const ruleStatus = await ruleStatusClient.getCurrentStatus({
             ruleId: rule.id,
             spaceId: context.securitySolution.getSpaceId(),
           });
           await deleteRules({
+            ruleId: rule.id,
             rulesClient,
             ruleStatusClient,
-            ruleStatuses,
-            id: rule.id,
           });
           return transformValidateBulkError(
             idOrRuleIdOrUnknown,
             rule,
-            ruleStatuses,
+            ruleStatus,
             isRuleRegistryEnabled
           );
         } catch (err) {

diff --git a/...ins/security_solution/server/lib/detection_engine/routes/rules/delete_rules_route.test.ts b/...ins/security_solution/server/lib/detection_engine/routes/rules/delete_rules_route.test.ts
@@ -12,7 +12,7 @@ import {
   getDeleteRequest,
   getFindResultWithSingleHit,
   getDeleteRequestById,
-  getRuleExecutionStatuses,
+  getRuleExecutionStatusSucceeded,
   getEmptySavedObjectsResponse,
 } from '../__mocks__/request_responses';
 import { requestContextMock, serverMock, requestMock } from '../__mocks__';
@@ -32,7 +32,9 @@ describe.each([
 
     clients.rulesClient.find.mockResolvedValue(getFindResultWithSingleHit(isRuleRegistryEnabled));
     clients.savedObjectsClient.find.mockResolvedValue(getEmptySavedObjectsResponse());
-    clients.ruleExecutionLogClient.find.mockResolvedValue(getRuleExecutionStatuses());
+    clients.ruleExecutionLogClient.getCurrentStatus.mockResolvedValue(
+      getRuleExecutionStatusSucceeded()
+    );
 
     deleteRulesRoute(server.router, isRuleRegistryEnabled);
   });

diff --git a/.../plugins/security_solution/server/lib/detection_engine/routes/rules/delete_rules_route.ts b/.../plugins/security_solution/server/lib/detection_engine/routes/rules/delete_rules_route.ts
@@ -62,18 +62,16 @@ export const deleteRulesRoute = (
           });
         }
 
-        const ruleStatuses = await ruleStatusClient.find({
-          logsCount: 6,
+        const currentStatus = await ruleStatusClient.getCurrentStatus({
           ruleId: rule.id,
           spaceId: context.securitySolution.getSpaceId(),
         });
         await deleteRules({
+          ruleId: rule.id,
           rulesClient,
           ruleStatusClient,
-          ruleStatuses,
-          id: rule.id,
         });
-        const transformed = transform(rule, ruleStatuses[0], isRuleRegistryEnabled);
+        const transformed = transform(rule, currentStatus, isRuleRegistryEnabled);
         if (transformed == null) {
           return siemResponse.error({ statusCode: 500, body: 'failed to transform alert' });
         } else {