Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Observability Assistant] Recall Bug in 8.12.2 / 8.13.0 Due to Lenient Prompt and Strict Parsing #177855

Closed
LucaWintergerst opened this issue Feb 29, 2024 · 0 comments · Fixed by #177898
Closed

[Observability Assistant] Recall Bug in 8.12.2 / 8.13.0 Due to Lenient Prompt and Strict Parsing #177855

LucaWintergerst opened this issue Feb 29, 2024 · 0 comments · Fixed by #177898
Labels
bug Fixes for quality problems that affect the customer experience Team:Obs AI Assistant Observability AI Assistant

Comments

@LucaWintergerst
Copy link
Contributor

Kibana version: 8.12.2 / 8.13.0

We've identified a recall bug within the Observability AI Assistant, specifically in versions 8.12.2 and 8.13.0, which affects the handling of scores by the context function https://github.com/elastic/kibana/blob/main/x-pack/plugins/observability_solution/observability_ai_assistant/server/functions/context.ts#L274-L278.

The core issue lies in the discrepancy between the leniency of the prompt and the strictness of the parsing mechanism.

Problem Description:
The expected format for scores returned by the Language Learning Model (LLM) is as follows:

index,score
index,score
index,score

However, we have observed two additional representations being returned occasionally, especially when using gpt-4-turbo:

A single line with scores separated by spaces (observed in 50% of the last 20 attempts)
Scores delimited by semicolons (observed in 10% of the last 20 attempts)
Example of observed formats:

0,0 1,0 2,0 3,0 4,0 5,0
0,1;1,1;2,7;3,0;4,0

We should explore:

  • changing the prompt to make it more robust
  • be more lenient in parsing the response
  • send all docs if the parsing fails (better to have less relevant knowledge than none at all)
@LucaWintergerst LucaWintergerst added the bug Fixes for quality problems that affect the customer experience label Feb 29, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Feb 29, 2024
dgieselaar added a commit to dgieselaar/kibana that referenced this issue Mar 2, 2024
@dgieselaar
[Obs AI Assistant] More lenient parsing of suggestion scores
8d8be64 
Closes elastic#177855. Also adds the scores to `data` for easier debugging.
@dgieselaar dgieselaar added the Team:Obs AI Assistant Observability AI Assistant label Mar 2, 2024
@botelastic botelastic bot removed the needs-team Issues missing a team label label Mar 2, 2024
@dgieselaar dgieselaar mentioned this issue Mar 2, 2024
Merged
dgieselaar added a commit that referenced this issue Mar 4, 2024
@dgieselaar
[Obs AI Assistant] More lenient parsing of suggestion scores (#177898)
0fd880b 
Closes #177855. Also adds the scores to `data` for easier debugging.
semd pushed a commit to semd/kibana that referenced this issue Mar 4, 2024
@dgieselaar @semd
[Obs AI Assistant] More lenient parsing of suggestion scores (elastic…
aae015f 
…#177898)

Closes elastic#177855. Also adds the scores to `data` for easier debugging.
fkanout pushed a commit to fkanout/kibana that referenced this issue Mar 4, 2024
@dgieselaar @fkanout
[Obs AI Assistant] More lenient parsing of suggestion scores (elastic…
70c0b23 
…#177898)

Closes elastic#177855. Also adds the scores to `data` for easier debugging.
dgieselaar added a commit to dgieselaar/kibana that referenced this issue Mar 6, 2024
@dgieselaar
[Obs AI Assistant] More lenient parsing of suggestion scores (elastic…
7ed5802 
…#177898)

Closes elastic#177855. Also adds the scores to `data` for easier debugging.

(cherry picked from commit 0fd880b)

# Conflicts:
#	x-pack/plugins/observability_ai_assistant/server/functions/parse_suggestion_scores.test.ts
#	x-pack/plugins/observability_ai_assistant/server/functions/parse_suggestion_scores.ts
dgieselaar added a commit that referenced this issue Mar 6, 2024
@dgieselaar
[8.13] [Obs AI Assistant] More lenient parsing of suggestion scores (#…
7b50e93 
…177898) (#178152)

# Backport

This will backport the following commits from `main` to `8.13`:
- [[Obs AI Assistant] More lenient parsing of suggestion scores
(#177898)](#177898)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Dario
Gieselaar","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-03-04T08:39:26Z","message":"[Obs
AI Assistant] More lenient parsing of suggestion scores
(#177898)\n\nCloses #177855. Also adds the scores to `data` for easier
debugging.","sha":"0fd880be2ba63da78505465a999f3102f1ded714","branchLabelMapping":{"^v8.14.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v8.13.0","v8.14.0","v8.12.3"],"number":177898,"url":"https://github.com/elastic/kibana/pull/177898","mergeCommit":{"message":"[Obs
AI Assistant] More lenient parsing of suggestion scores
(#177898)\n\nCloses #177855. Also adds the scores to `data` for easier
debugging.","sha":"0fd880be2ba63da78505465a999f3102f1ded714"}},"sourceBranch":"main","suggestedTargetBranches":["8.13","8.12"],"targetPullRequestStates":[{"branch":"8.13","label":"v8.13.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.14.0","labelRegex":"^v8.14.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/177898","number":177898,"mergeCommit":{"message":"[Obs
AI Assistant] More lenient parsing of suggestion scores
(#177898)\n\nCloses #177855. Also adds the scores to `data` for easier
debugging.","sha":"0fd880be2ba63da78505465a999f3102f1ded714"}},{"branch":"8.12","label":"v8.12.3","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
dgieselaar added a commit to dgieselaar/kibana that referenced this issue Mar 7, 2024
@dgieselaar
[Obs AI Assistant] More lenient parsing of suggestion scores (elastic…
8f0efe2 
…#177898)

Closes elastic#177855. Also adds the scores to `data` for easier debugging.

(cherry picked from commit 0fd880b)

# Conflicts:
#	x-pack/plugins/observability_ai_assistant/server/functions/parse_suggestion_scores.test.ts
#	x-pack/plugins/observability_ai_assistant/server/functions/parse_suggestion_scores.ts
#	x-pack/plugins/observability_ai_assistant/server/functions/recall.ts
dgieselaar added a commit that referenced this issue Mar 7, 2024
@dgieselaar
[8.12] [Obs AI Assistant] More lenient parsing of suggestion scores (#…
9dc1ed1 
…177898) (#178178)

# Backport

This will backport the following commits from `main` to `8.12`:
- [[Obs AI Assistant] More lenient parsing of suggestion scores
(#177898)](#177898)

<!--- Backport version: 8.9.8 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Dario
Gieselaar","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-03-04T08:39:26Z","message":"[Obs
AI Assistant] More lenient parsing of suggestion scores
(#177898)\n\nCloses #177855. Also adds the scores to `data` for easier
debugging.","sha":"0fd880be2ba63da78505465a999f3102f1ded714","branchLabelMapping":{"^v8.14.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","v8.13.0","v8.14.0","v8.12.3"],"number":177898,"url":"https://github.com/elastic/kibana/pull/177898","mergeCommit":{"message":"[Obs
AI Assistant] More lenient parsing of suggestion scores
(#177898)\n\nCloses #177855. Also adds the scores to `data` for easier
debugging.","sha":"0fd880be2ba63da78505465a999f3102f1ded714"}},"sourceBranch":"main","suggestedTargetBranches":["8.12"],"targetPullRequestStates":[{"branch":"8.13","label":"v8.13.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"url":"https://github.com/elastic/kibana/pull/178152","number":178152,"state":"OPEN"},{"branch":"main","label":"v8.14.0","labelRegex":"^v8.14.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/177898","number":177898,"mergeCommit":{"message":"[Obs
AI Assistant] More lenient parsing of suggestion scores
(#177898)\n\nCloses #177855. Also adds the scores to `data` for easier
debugging.","sha":"0fd880be2ba63da78505465a999f3102f1ded714"}},{"branch":"8.12","label":"v8.12.3","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Fixes for quality problems that affect the customer experience Team:Obs AI Assistant Observability AI Assistant
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Obs AI Assistant] More lenient parsing of suggestion scores dgieselaar/kibana
2 participants
@dgieselaar @LucaWintergerst