-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[Security Solution] [Elastic AI Assistant] Retrieval Augmented Genera…
…tion (RAG) for Alerts (#172542) ## [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts This PR implements _Retrieval Augmented Generation_ (RAG) for Alerts in the Security Solution. This feature enables users to ask the assistant questions about the latest and riskiest open alerts in their environment using natural language, for example: - _How many alerts are currently open?_ - _Which alerts should I look at first?_ - _Did we have any alerts with suspicious activity on Windows machines?_ ### More context Previously, the assistant relied solely on the knowledge of the configured LLM and _singular_ alerts or events passed _by the client_ to the LLM as prompt context. This new feature: - Enables _multiple_ alerts to be passed by the _server_ as context to the LLM, via [LangChain tools](#167097) - Applies the user's [anonymization](#159857) settings to those alerts - Only fields allowed by the user will be sent as context to the LLM - Users may enable or disable anonymization for specific fields (via settings) - Click the conversation's `Show anonymized` toggle to see the anonymized values sent to / received from the LLM:  ### Settings This feature is enabled and configured via the `Knowledge Base` > `Alerts` settings in the screenshot below:  - The `Alerts` toggle enables or disables the feature - The slider has a range of `10` - `100` alerts (default: `20`) When the setting above is enabled, up to `n` alerts (as determined by the slider) that meet the following criteria will be returned: - the `kibana.alert.workflow_status` must be `open` - the alert must have been generated in the last `24 hours` - the alert must NOT be a `kibana.alert.building_block_type` alert - the `n` alerts are ordered by `kibana.alert.risk_score`, to prioritize the riskiest alerts ### Feature flag To use this feature: 1) Add the `assistantRagOnAlerts` feature flag to the `xpack.securitySolution.enableExperimental` setting in `config/kibana.yml` (or `config/kibana.dev.yml` in local development environments), per the example below: ``` xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts'] ``` 2) Enable the `Alerts` toggle in the Assistant's `Knowledge Base` settings, per the screenshot below:  ## How it works - When the `Alerts` settings toggle is enabled, http `POST` requests to the `/internal/elastic_assistant/actions/connector/{id}/_execute` route include the following new (optional) parameters: - `alertsIndexPattern`, the alerts index for the current Kibana Space, e.g. `.alerts-security.alerts-default` - `allow`, the user's `Allowed` fields in the `Anonymization` settings, e.g. `["@timestamp", "cloud.availability_zone", "file.name", "user.name", ...]` - `allowReplacement`, the user's `Anonymized` fields in the `Anonymization` settings, e.g. `["cloud.availability_zone", "host.name", "user.name", ...]` - `replacements`, a `Record<string, string>` of replacements (generated on the server) that starts empty for a new conversation, and accumulates anonymized values until the conversation is cleared, e.g. ```json "replacements": { "e4f935c0-5a80-47b2-ac7f-816610790364": "Host-itk8qh4tjm", "cf61f946-d643-4b15-899f-6ffe3fd36097": "rpwmjvuuia", "7f80b092-fb1a-48a2-a634-3abc61b32157": "6astve9g6s", "f979c0d5-db1b-4506-b425-500821d00813": "Host-odqbow6tmc", // ... }, ``` - `size`, the numeric value set by the slider in the user's `Knowledge Base > Alerts` setting, e.g. `20` - The `postActionsConnectorExecuteRoute` function in `x-pack/plugins/elastic_assistant/server/routes/post_actions_connector_execute.ts` was updated to accept the new optional parameters, and to return an updated `replacements` with every response. (Every new request that is processed on the server may add additional anonymized values to the `replacements` returned in the response.) - The `callAgentExecutor` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/execute_custom_llm_chain/index.ts` previously used a hard-coded array of LangChain tools that had just one entry, for the `ESQLKnowledgeBaseTool` tool. That hard-coded array was replaced in this PR with a call to the (new) `getApplicableTools` function: ```typescript const tools: Tool[] = getApplicableTools({ allow, allowReplacement, alertsIndexPattern, assistantLangChain, chain, esClient, modelExists, onNewReplacements, replacements, request, size, }); ``` - The `getApplicableTools` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/index.ts` examines the parameters in the `KibanaRequest` and only returns a filtered set of LangChain tools. If the request doesn't contain all the parameters required by a tool, it will NOT be returned by `getApplicableTools`. For example, if the required anonymization parameters are not included in the request, the `open-alerts` tool will not be returned. - The new `alert-counts` LangChain tool returned by the `getAlertCountsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_tool.ts` provides the LLM the results of an aggregation on the last `24` hours of alerts (in the current Kibana Space), grouped by `kibana.alert.severity`. See the `getAlertsCountQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/alert_counts/get_alert_counts_query.ts` for details - The new `open-alerts` LangChain tool returned by the `getOpenAlertsTool` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_tool.ts` provides the LLM up to `size` non-building-block alerts generated in the last `24` hours (in the current Kibana Space) with an `open` workflow status, ordered by `kibana.alert.risk_score` to prioritize the riskiest alerts. See the `getOpenAlertsQuery` function in `x-pack/plugins/elastic_assistant/server/lib/langchain/tools/open_alerts/get_open_alerts_query.ts` for details. - On the client, a conversation continues to accumulate additional `replacements` (and send them in subsequent requests) until the conversation is cleared - Anonymization functions that were only invoked by the browser were moved from the (browser) `kbn-elastic-assistant` package in `x-pack/packages/kbn-elastic-assistant/` to a new common package: `x-pack/packages/kbn-elastic-assistant-common` - The new `kbn-elastic-assistant-common` package is also consumed by the `elastic_assistant` (server) plugin: `x-pack/plugins/elastic_assistant`
- Loading branch information
1 parent
9695939
commit 3f0fa7d
Showing
81 changed files
with
5,656 additions
and
124 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Validating CODEOWNERS rules …
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # @kbn/elastic-assistant-common | ||
|
|
||
| This package provides common code consumed in both the browser, i.e. the | ||
| `packages/kbn-elastic-assistant` package, and on the server, i.e. the | ||
| `plugins/elastic_assistant` plugin. | ||
|
|
||
| For example, the data anonymization functions exported by this package | ||
| are be used in both the browser, and on the server. | ||
|
|
||
| ## Maintainers | ||
|
|
||
| Maintained by the Security Solution team | ||
|
|
||
| ## Running unit tests with code coverage | ||
|
|
||
| To (interactively) run unit tests with code coverage, run the following command: | ||
|
|
||
| ```sh | ||
| cd $KIBANA_HOME && node scripts/jest --watch x-pack/packages/kbn-elastic-assistant-common --coverage | ||
| ``` |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
45 changes: 45 additions & 0 deletions
45
...s/kbn-elastic-assistant-common/impl/data_anonymization/get_anonymized_value/index.test.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import { invert } from 'lodash/fp'; | ||
|
|
||
| import { getAnonymizedValue } from '.'; | ||
|
|
||
| jest.mock('uuid', () => ({ | ||
| v4: () => 'test-uuid', | ||
| })); | ||
|
|
||
| describe('getAnonymizedValue', () => { | ||
| beforeEach(() => jest.clearAllMocks()); | ||
|
|
||
| it('returns a new UUID when currentReplacements is not provided', () => { | ||
| const currentReplacements = undefined; | ||
| const rawValue = 'test'; | ||
|
|
||
| const result = getAnonymizedValue({ currentReplacements, rawValue }); | ||
|
|
||
| expect(result).toBe('test-uuid'); | ||
| }); | ||
|
|
||
| it('returns an existing anonymized value when currentReplacements contains an entry for it', () => { | ||
| const rawValue = 'test'; | ||
| const currentReplacements = { anonymized: 'test' }; | ||
| const rawValueToReplacement = invert(currentReplacements); | ||
|
|
||
| const result = getAnonymizedValue({ currentReplacements, rawValue }); | ||
| expect(result).toBe(rawValueToReplacement[rawValue]); | ||
| }); | ||
|
|
||
| it('returns a new UUID with currentReplacements if no existing match', () => { | ||
| const rawValue = 'test'; | ||
| const currentReplacements = { anonymized: 'other' }; | ||
|
|
||
| const result = getAnonymizedValue({ currentReplacements, rawValue }); | ||
|
|
||
| expect(result).toBe('test-uuid'); | ||
| }); | ||
| }); |
26 changes: 26 additions & 0 deletions
26
...ckages/kbn-elastic-assistant-common/impl/data_anonymization/get_anonymized_value/index.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import { invert } from 'lodash/fp'; | ||
| import { v4 } from 'uuid'; | ||
|
|
||
| export const getAnonymizedValue = ({ | ||
| currentReplacements, | ||
| rawValue, | ||
| }: { | ||
| currentReplacements: Record<string, string> | undefined; | ||
| rawValue: string; | ||
| }): string => { | ||
| if (currentReplacements != null) { | ||
| const rawValueToReplacement: Record<string, string> = invert(currentReplacements); | ||
| const existingReplacement: string | undefined = rawValueToReplacement[rawValue]; | ||
|
|
||
| return existingReplacement != null ? existingReplacement : v4(); | ||
| } | ||
|
|
||
| return v4(); | ||
| }; |
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
97 changes: 97 additions & 0 deletions
97
x-pack/packages/kbn-elastic-assistant-common/impl/data_anonymization/helpers/index.test.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| import { isAllowed, isAnonymized, isDenied, getIsDataAnonymizable } from '.'; | ||
|
|
||
| describe('helpers', () => { | ||
| beforeEach(() => jest.clearAllMocks()); | ||
|
|
||
| describe('getIsDataAnonymizable', () => { | ||
| it('returns false for string data', () => { | ||
| const rawData = 'this will not be anonymized'; | ||
|
|
||
| const result = getIsDataAnonymizable(rawData); | ||
|
|
||
| expect(result).toBe(false); | ||
| }); | ||
|
|
||
| it('returns true for key / values data', () => { | ||
| const rawData = { key: ['value1', 'value2'] }; | ||
|
|
||
| const result = getIsDataAnonymizable(rawData); | ||
|
|
||
| expect(result).toBe(true); | ||
| }); | ||
| }); | ||
|
|
||
| describe('isAllowed', () => { | ||
| it('returns true when the field is present in the allowSet', () => { | ||
| const allowSet = new Set(['fieldName1', 'fieldName2', 'fieldName3']); | ||
|
|
||
| expect(isAllowed({ allowSet, field: 'fieldName1' })).toBe(true); | ||
| }); | ||
|
|
||
| it('returns false when the field is NOT present in the allowSet', () => { | ||
| const allowSet = new Set(['fieldName1', 'fieldName2', 'fieldName3']); | ||
|
|
||
| expect(isAllowed({ allowSet, field: 'nonexistentField' })).toBe(false); | ||
| }); | ||
| }); | ||
|
|
||
| describe('isDenied', () => { | ||
| it('returns true when the field is NOT in the allowSet', () => { | ||
| const allowSet = new Set(['field1', 'field2']); | ||
| const field = 'field3'; | ||
|
|
||
| expect(isDenied({ allowSet, field })).toBe(true); | ||
| }); | ||
|
|
||
| it('returns false when the field is in the allowSet', () => { | ||
| const allowSet = new Set(['field1', 'field2']); | ||
| const field = 'field1'; | ||
|
|
||
| expect(isDenied({ allowSet, field })).toBe(false); | ||
| }); | ||
|
|
||
| it('returns true for an empty allowSet', () => { | ||
| const allowSet = new Set<string>(); | ||
| const field = 'field1'; | ||
|
|
||
| expect(isDenied({ allowSet, field })).toBe(true); | ||
| }); | ||
|
|
||
| it('returns false when the field is an empty string and allowSet contains the empty string', () => { | ||
| const allowSet = new Set(['', 'field1']); | ||
| const field = ''; | ||
|
|
||
| expect(isDenied({ allowSet, field })).toBe(false); | ||
| }); | ||
| }); | ||
|
|
||
| describe('isAnonymized', () => { | ||
| const allowReplacementSet = new Set(['user.name', 'host.name']); | ||
|
|
||
| it('returns true when the field is in the allowReplacementSet', () => { | ||
| const field = 'user.name'; | ||
|
|
||
| expect(isAnonymized({ allowReplacementSet, field })).toBe(true); | ||
| }); | ||
|
|
||
| it('returns false when the field is NOT in the allowReplacementSet', () => { | ||
| const field = 'foozle'; | ||
|
|
||
| expect(isAnonymized({ allowReplacementSet, field })).toBe(false); | ||
| }); | ||
|
|
||
| it('returns false when allowReplacementSet is empty', () => { | ||
| const emptySet = new Set<string>(); | ||
| const field = 'user.name'; | ||
|
|
||
| expect(isAnonymized({ allowReplacementSet: emptySet, field })).toBe(false); | ||
| }); | ||
| }); | ||
| }); |
23 changes: 23 additions & 0 deletions
23
x-pack/packages/kbn-elastic-assistant-common/impl/data_anonymization/helpers/index.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License | ||
| * 2.0; you may not use this file except in compliance with the Elastic License | ||
| * 2.0. | ||
| */ | ||
|
|
||
| export const getIsDataAnonymizable = (rawData: string | Record<string, string[]>): boolean => | ||
| typeof rawData !== 'string'; | ||
|
|
||
| export const isAllowed = ({ allowSet, field }: { allowSet: Set<string>; field: string }): boolean => | ||
| allowSet.has(field); | ||
|
|
||
| export const isDenied = ({ allowSet, field }: { allowSet: Set<string>; field: string }): boolean => | ||
| !allowSet.has(field); | ||
|
|
||
| export const isAnonymized = ({ | ||
| allowReplacementSet, | ||
| field, | ||
| }: { | ||
| allowReplacementSet: Set<string>; | ||
| field: string; | ||
| }): boolean => allowReplacementSet.has(field); |
Oops, something went wrong.