[Security Solution] [Elastic AI Assistant] Include acknowledged alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts)

[andrew-goldstein]

[Security Solution] [Elastic AI Assistant] Include acknowledged alerts in the context sent to the LLM (Retrieval Augmented Generation (RAG) for Alerts)

This PR updates the query used by [Security Solution] [Elastic AI Assistant] Retrieval Augmented Generation (RAG) for Alerts #172542 to include alerts with a kibana.alert.workflow_status value of acknowledged.

The query previously only returned alerts with a status of open. This change ensures both open and acknowledged alerts are provided as context to the LLM.

Updated Anonymization defaults

Three fields, detailed below, were added as anonymization defaults because they improve the quality of responses from the LLM when it answers questions about alerts.

For example, the LLM can refer to specific alerts by ID when the _id field is provided.

This PR makes the following additive changes to the Assistant's Anonymization defaults:

Field	Allow by default	Anonymize by default	Value add
_id	✅	✅	An anonymized _id field enables responses from the LLM to refer to specific documents (but doesn't provide it the actual document IDs).
kibana.alert.risk_score	✅	❌	The getOpenAndAcknowledgedAlertsQuery query sorts alerts by kibana.alert.risk_score to return the n riskiest alerts. Allowing this field (by default) enables the LLM to include actual alert risk scores in responses.
kibana.alert.workflow_status	✅	❌	The getOpenAndAcknowledgedAlertsQuery query filters alerts by kibana.alert.workflow_status to ensure only open and acknowledged alerts are provided as context to the LLM. Allowing this field (by default) enables the LLM answer questions about workflow status, and echo the workflow status of alerts in responses.

• Clicking the Reset button shown in the screenshot below will reset the user's Anonymization defaults, such that they include the additive changes in the table above:

Updated settings text

The text in the settings below was also updated:

Desk testing

To desk test this change:

• Enable the assistantRagOnAlerts feature flag described in #172542 must be enabled, per the following example:

xpack.securitySolution.enableExperimental: ['assistantRagOnAlerts']

• The Alerts feature must be enabled in the assistant settings, per the screenshot below:

1. Navigate to Security > Alerts

2. Click the AI Assistant button to open the assistant

3. Click the Settings gear to open the assistant settings

4. Click the Anonymization category

5. Click the Reset button, shown in the screenshot below

Expected results

• 65 fields are allowed by default, per the screenshot above
• 12 fields are anonymized by default, per the screenshot above
• The _id field is allowed by default, per the screenshot above
• The _id field is anonymized by default, per the screenshot above

6. Type kibana.alert.risk in the search box

Expected result

• The kibana.alert.risk_score field is allowed by default

7. Type kibana.alert.workflow in the search box

Expected result

• The kibana.alert.workflow_status field is allowed by default

8. Click Save

9. Click the X button to clear the conversation

10. Close the assistant

11. Add the following two fields as columns to the Alerts page table:

• kibana.alert.workflow_status
• _id

12. Sort the table, first by kibana.alert.risk_score from high to low, and then by @timestamp from new to old, per the screenshot below:

13. Filter the alerts page to only show open and acknowledged alerts

Expected result

• The alerts page has custom columns, sorting, and filtering, per the screenshot below:

14. Click the AI Assistant button to open the assistant

15. Ask the assistant:

What is the workflow status of my alerts?

Expected result

• The assistant will report on the workflow status of alerts, per the example response below:

The workflow status for your alerts is currently 'open'. This status was observed on alerts related to processes started by Mimikatz, a known tool used in many cyberattacks, and sequences of processes that are often indicative of malicious activity. The severity of most of these alerts is 'high' or 'critical'. You may want to investigate these alerts further to ensure there's no ongoing threat to your system.

16. Close the assistant

17. Change the workflow status of an alert in the Alerts table from open to acknowledged

Expected result

• The alerts table shows the updated alert, per the screenshot below:

18. Once again, open the assistant

19. Once again, ask the (same) question:

What is the workflow status of my alerts?

Expected result

• The response from the assistant makes reference to the alert who's workflow status was changed from open to acknowledged, per the example response below:

Based on the latest information, your alerts mainly show 'open' status, indicating that they have not been resolved yet. Some alerts have been acknowledged. Most of these unaddressed alerts have a critical severity rating and are primarily triggered by a Mimikatz process start and an EQL process sequence. You may want to prioritize these if the severity of the threat they pose is truly high or critical. It's also noteworthy that some alerts have a high severity rating. You should review all of these alerts as soon as possible to ensure your systems are secure.

20. Ask the assistant for details about the acknowledged alerts:

What are the details of the acknowledged alerts?

Expected result

The assistant for details about the acknowledged alert that, for example, includes the kibana.alert.risk_score, per the example response below:

In response to your previous question, here are the details of the acknowledged alerts:

1. There is a 'mimikatz process started' alert, which is of 'critical' severity and 'acknowledged' status. It has a high risk score of 99. Its threat tactic is 'Command and Control'. The process involved was 'mimikatz.exe' running with arguments '--fo1'.

2. A 'Threshold rule' alert of 'critical' severity and 'open' status has also been detected with a risk score of 99 and threat tactic 'Collection'.

3. Lastly, there are several 'EQL process sequence' alerts of 'high' severity with 'open' status. These alerts involve execution of various processes including 'mimikatz.exe', 'lsass.exe', and 'notepad.exe'. Risk score for these alerts is 73 and the threat tactic involved is 'Execution'.

Please, take appropriate action to address these alerts.

21. Ask the assistant for the _id of the acknowledged alert:

What is the id of the acknowledged alert?

Expected results

• The response from the assistant contains the _id of the acknowledged alert, per the example response below:

The id of the acknowledged alert is 'db9e3dbaf40a37e3b7b95d8015e99c5721b416731e04b9140536675f6e4fd170'. This alert was for a 'mimikatz process started' event with a severity rating of 'critical' and a risk score of 99. The host name associated with this alert is 'Host-terkvbzvtj'.

• The _id shown in the assistant is the same _id of the acknowledged alert on the alerts page, per the screeenshot below:

22. Click the Show anonymized toggle in the assistant

Expected result

• The _id shown in the latest result is replaced with the actual anonymized value that was sent to the LLM, per the example screenshot below:

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[spong]

Updated queries to include acknowledged alerts LGTM 👍

[logeekal]

@andrew-goldstein
nit: does it make sense to rename folder and file name from open_alerts to eligible_alerts or something else?

[andrew-goldstein]

@andrew-goldstein nit: does it make sense to rename folder and file name from open_alerts to eligible_alerts or something else?

Thanks @logeekal!

• I implemented your feedback in this commit

• The name eligible_alerts felt slightly out context as a folder and file name, perhaps because it wasn't clear what eligible meant when looking at the file system. Based purely on feels, and with the aim of avoiding the introduction of a "new" concept like eligible , the renamed function and folders / files now look like this (when imported):

Before:

import { getOpenAlertsTool } from './open_alerts/get_open_alerts_tool';

After:

import { getOpenAndAcknowledgedAlertsTool } from './open_and_acknowledged_alerts/get_open_and_acknowledged_alerts_tool';

Does that feel like a reasonable change based on the feedback?

[logeekal]

Yes, I agree 100%. this change makes sense. Thanks for implementing it.

[andrew-goldstein]

Field	Allow by default	Anonymize by default	Value add
_id	✅	✅	An anonymized _id field enables responses from the LLM to refer to specific documents (but doesn't provide it the actual document IDs).
kibana.alert.risk_score	✅	❌	The getOpenAndAcknowledgedAlertsQuery query sorts alerts by kibana.alert.risk_score to return the n riskiest alerts. Allowing this field (by default) enables the LLM to include actual alert risk scores in responses.
kibana.alert.workflow_status	✅	❌	The getOpenAndAcknowledgedAlertsQuery query filters alerts by kibana.alert.workflow_status to ensure only open and acknowledged alerts are provided as context to the LLM. Allowing this field (by default) enables the LLM answer questions about workflow status, and echo the workflow status of alerts in responses.

[logeekal]

@andrew-goldstein ,

I attempted to do the Desk testing but I got stuck with below error. Could it be because of the token I have? I am not sure.

Additionally, do you think below text should also be revised, since we are not looking at Open alerts only. Sorry for not noticing it last time.

[andrew-goldstein]

@andrew-goldstein ,

I attempted to do the Desk testing but I got stuck with below error. Could it be because of the token I have? I am not sure.

The error in the first screenshot happened because the number of alerts sent as context to the LLM exceeded the model's maximum context length.

• The maximum context length will vary from model to model
  • Users may configure multiple connectors and work with multiple models
• The total number of fields sent per alert (as context) will also vary
  • The user's anonymization settings, which determine which fields will be sent, will vary
  • The fields present in the alerts will vary; some alerts may have more fields, some will have fewer fields

Given the variability above, a slider is provided so users may adjust the total number of alerts sent as context.

The default value for the slider is 20. We expect to increase this default over time.

It appears in the 2nd screenshot that you may have increased the default.

• Would you be willing to retry with the default value of 20?
• As a result of the error you reported above, we added additional guidance: Select fewer alerts if the model's maximum context length is frequently exceeded. to the Alerts settings, as detailed below

Additionally, do you think below text should also be revised, since we are not looking at Open alerts only. Sorry for not noticing it last time.

Thanks for noting that this text should also be revised, because the query was updated to include acknowledged alerts.

• It now reads: Ask questions about the (n) latest and riskiest open and acknowledged alerts in your environment.
• Additional text Select fewer alerts if the model's maximum context length is frequently exceeded. was added to guide users selecting a lower value for n when they encounter context length limits

The updates are highlighted in the screenshot below:

[logeekal]

Desk Testing work perfectly. I was able to follow the steps mentioned and all of them passed.

I also tried changing the sorting of data in Alert Table and it changed the response from Assistant perfectly.

Great work 🚀 .

One point to note:

I hit the context limit event with 20 alerts so I had to reduce to 10. For future enhancement, I think we should instruct users on what to do for known errors such as below. I guess action will be different in different type of Conversations. Not sure if it is even feasible.

An error occurred sending your message.

ActionsClientLlm: action result status is error: an error occurred while running the action - Status code: 400. Message: API Error: model_error - This model's maximum context length is 8192 tokens. However, your messages resulted in 8996 tokens. Please reduce the length of the messages.

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 0982303

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	13.1MB	13.1MB	+3.0KB

History

• 💛 Build #183412 was flaky c0a29ab1f76fe25e2bb8b421cddfd68f880ee206
• 💚 Build #183321 succeeded 322707ec682dc9c5a94ddc1f68bd2a27fe011fd7
• 💛 Build #183062 was flaky a92be36a8a39d1600a5bde1b37a3ceb480dd53fb

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @andrew-goldstein

[kibanamachine]

💚 All backports created successfully

Status	Branch	Result
✅	8.12

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

[spong]

nit: open an acknowledged

Found in managing the merge w/ #172234. I got this 👍