Skip to content

Commit

Permalink
[Obs AI Assistant] Include an AdHoc instruction about the slack conne…
Browse files Browse the repository at this point in the history
…ctor to avoid executing a loop (elastic#199531)

Closes elastic#185028

## Summary

### Problem
The ticket mentions that there was an issue with displaying results.
However both `display results` and `visualize query` are working as
expected based on my investigation. More details including a video are
attached
[here](elastic#185028 (comment)).

The function calling loop seems to occur when the AI Assistant is trying
to send the output to Slack via the Kibana Slack connector. In order to
do this, the LLM invokes the function `execute_connector`. For the Slack
connector, `id` and `params` properties are required. However, the LLM
only populated `id` and not `params` which causes an error when
validated against the schema for the Slack connector.

- Sometimes, it's able to retry a few times and successfully send the
output to Slack.
- Sometimes, it goes into a loop trying to find `params` and failing
repeatedly.

Attaching another similar issue for more context -
elastic#195564

_With the solution below, my intention is to send the output to Slack in
one go, without retrying the function `execute_connector`._

### Solution
Based on the solutions I experimented with, seems like we need to force
the LLM to understand what's needed for the Slack connector. I tried a
few options here and the combination of updates that worked are as
follows:
- Appending an AdHoc instruction about the Slack connector properties.
(This gives the LLM some additional information about the required
properties)
- Updating the `properties` attached to the connector (when passing the
connector list to the LLM), to reflect both `id` and `params` with
`message`.

With the above change, the AI Assistant has managed to consistently send
the output to Slack _**without any retries**_ because of missing
`params`.

### Screenshots:
(all alert triggers successfully sent the output to Slack without having
to retry the function)

<img width="1452" alt="success-attempts-to-slack-connector"
src="https://github.com/user-attachments/assets/715a5957-2c04-4a55-868f-34abe564f6d4">
  • Loading branch information
viduni94 authored and CAWilson94 committed Nov 18, 2024
1 parent 803384b commit 7f0bee6
Show file tree
Hide file tree
Showing 2 changed files with 34 additions and 3 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,15 @@
import { FunctionRegistrationParameters } from '.';
import { FunctionVisibility } from '../../common';

export const EXECUTE_CONNECTOR_FUNCTION_NAME = 'execute_connector';

export function registerExecuteConnectorFunction({
functions,
resources,
}: FunctionRegistrationParameters) {
functions.registerFunction(
{
name: 'execute_connector',
name: EXECUTE_CONNECTOR_FUNCTION_NAME,
description: 'Use this function when user explicitly asks to call a kibana connector.',
visibility: FunctionVisibility.AssistantOnly,
parameters: {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ import {
JiraParamsSchema,
PagerdutyParamsSchema,
SlackApiParamsSchema,
SlackParamsSchema,
WebhookParamsSchema,
} from '@kbn/stack-connectors-plugin/server';
import { ObservabilityAIAssistantRouteHandlerResources } from '@kbn/observability-ai-assistant-plugin/server/routes/types';
Expand All @@ -37,14 +36,26 @@ import { CompatibleJSONSchema } from '@kbn/observability-ai-assistant-plugin/com
import { AlertDetailsContextualInsightsService } from '@kbn/observability-plugin/server/services';
import { getSystemMessageFromInstructions } from '@kbn/observability-ai-assistant-plugin/server/service/util/get_system_message_from_instructions';
import { AdHocInstruction } from '@kbn/observability-ai-assistant-plugin/common/types';
import { EXECUTE_CONNECTOR_FUNCTION_NAME } from '@kbn/observability-ai-assistant-plugin/server/functions/execute_connector';
import { convertSchemaToOpenApi } from './convert_schema_to_open_api';
import { OBSERVABILITY_AI_ASSISTANT_CONNECTOR_ID } from '../../common/rule_connector';

const CONNECTOR_PRIVILEGES = ['api:observabilityAIAssistant', 'app:observabilityAIAssistant'];

const connectorParamsSchemas: Record<string, CompatibleJSONSchema> = {
'.slack': {
type: 'object',
properties: {
id: { type: 'string' },
params: {
type: 'object',
properties: {
message: { type: 'string' },
},
},
},
},
'.slack_api': convertSchemaToOpenApi(SlackApiParamsSchema),
'.slack': convertSchemaToOpenApi(SlackParamsSchema),
'.email': convertSchemaToOpenApi(EmailParamsSchema),
'.webhook': convertSchemaToOpenApi(WebhookParamsSchema),
'.jira': convertSchemaToOpenApi(JiraParamsSchema),
Expand Down Expand Up @@ -189,6 +200,24 @@ If available, include the link of the conversation at the end of your answer.`
),
};

const hasSlackConnector = !!connectorsList.filter(
(connector) => connector.actionTypeId === '.slack'
).length;

if (hasSlackConnector && functionClient.hasFunction(EXECUTE_CONNECTOR_FUNCTION_NAME)) {
const slackConnectorInstruction: AdHocInstruction = {
instruction_type: 'application_instruction',
text: dedent(
`The execute_connector function can be used to invoke Kibana connectors.
To send to the Slack connector, you need the following arguments:
- the "id" of the connector
- the "params" parameter that you will fill with the message
Please include both "id" and "params.message" in the function arguments when executing the Slack connector..`
),
};
functionClient.registerAdhocInstruction(slackConnectorInstruction);
}

const alertsContext = await getAlertsContext(
execOptions.params.rule,
execOptions.params.alerts,
Expand Down

0 comments on commit 7f0bee6

Please sign in to comment.