Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(chat): filter namespace messages from history if it exists in metadata VSCODE-611 #866

Merged
merged 60 commits into from
Nov 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
a54d67d
Filter long / invalid prompts
gagik Oct 29, 2024
6ff0d8d
Try helper include
gagik Oct 30, 2024
c9aa589
Use content value
gagik Oct 30, 2024
698051f
Use a cleaner test
gagik Oct 30, 2024
bf05722
delete old helper
gagik Oct 30, 2024
2445dc5
Add explainer
gagik Oct 30, 2024
ff0c1cc
WIP
gagik Oct 31, 2024
f9dd536
Move around dependencies
gagik Nov 1, 2024
09c7414
Remove grep
gagik Nov 1, 2024
aada8e1
Use firstCall
gagik Nov 1, 2024
eb2c7dc
Add test filtering
gagik Nov 1, 2024
bcd260b
Update CONTRIBUTING.md
gagik Nov 1, 2024
a9beef1
Escape the environment variable
gagik Nov 1, 2024
2e4bd70
Merge branch 'gagik/add-test-filtering' of github.com:mongodb-js/vsco…
gagik Nov 1, 2024
33c995e
Fix wording
gagik Nov 1, 2024
8582193
Add schema tests
gagik Nov 1, 2024
a8bc30d
align tests and use a stub
gagik Nov 1, 2024
5ddc7fb
Add saving to metadata
gagik Nov 1, 2024
06435e8
Move things
gagik Nov 3, 2024
5a58171
Better org
gagik Nov 3, 2024
fdbabfc
Merge branch 'gagik/add-test-filtering' of github.com:mongodb-js/vsco…
gagik Nov 3, 2024
f23e19f
simplify tests and picking logic
gagik Nov 3, 2024
274fe17
typos
gagik Nov 3, 2024
54885d6
Align with broken test
gagik Nov 4, 2024
cfb9d61
Add error info and tests
gagik Nov 4, 2024
f080016
wrap l10n
gagik Nov 4, 2024
190f4f6
Merge branch 'gagik/one-no-collection-handling' of github.com:mongodb…
gagik Nov 5, 2024
182dc15
wrap in l10n
gagik Nov 5, 2024
a341e45
remove settings change
gagik Nov 5, 2024
bec38d8
Merge branch 'gagik/no-database-or-collection-error' of github.com:mo…
gagik Nov 5, 2024
548f247
Merge branch 'gagik/fix-long-prompts' of github.com:mongodb-js/vscode…
gagik Nov 5, 2024
c1b6534
Merge branch 'main' of github.com:mongodb-js/vscode into gagik/no-dat…
gagik Nov 5, 2024
378b202
Merge branch 'main' of github.com:mongodb-js/vscode into gagik/one-no…
gagik Nov 5, 2024
f468adf
Add tests
gagik Nov 5, 2024
62c07ef
Apply suggestions from code review
gagik Nov 6, 2024
f2b54ea
changes from review
gagik Nov 6, 2024
276ce90
move error types to inside participant
gagik Nov 6, 2024
bf2a0c7
switch to parametrized tests
gagik Nov 6, 2024
02f45dc
Merge branch 'gagik/one-no-collection-handling' into gagik/no-databas…
gagik Nov 6, 2024
a1765ef
remove patched vscode
gagik Nov 6, 2024
ccacb8a
remove vscode
gagik Nov 6, 2024
886aa45
better comments
gagik Nov 6, 2024
52dbdf6
fix potential CI discrepancy
gagik Nov 7, 2024
686c7cd
Merge branch 'main' of github.com:mongodb-js/vscode into gagik/one-no…
gagik Nov 7, 2024
fee668e
combine message text
gagik Nov 7, 2024
e1d12bf
add explicit undefined returns
gagik Nov 7, 2024
c92cdcb
Merge branch 'gagik/one-no-collection-handling' into gagik/no-databas…
gagik Nov 7, 2024
985ed49
Move tests to parameterized
gagik Nov 7, 2024
f012622
Merge branch 'main' of github.com:mongodb-js/vscode into gagik/filter…
gagik Nov 7, 2024
aba650c
delete mistaken duplicate
gagik Nov 7, 2024
650bd79
cleanup tests
gagik Nov 7, 2024
0843c64
fix expected history
gagik Nov 7, 2024
6cb9ad3
revert
gagik Nov 7, 2024
0bbeeb2
Merge branch 'main' into gagik/filter-namespace
gagik Nov 8, 2024
377b6bd
Merge branch 'main' of github.com:mongodb-js/vscode into gagik/filter…
gagik Nov 8, 2024
853eccf
Merge branch 'main' of github.com:mongodb-js/vscode into gagik/no-dat…
gagik Nov 8, 2024
86a33f2
Merge branch 'gagik/no-database-or-collection-error' of github.com:mo…
gagik Nov 8, 2024
4ad4c5f
small cleanup
gagik Nov 8, 2024
0a8b085
Merge branch 'main' of github.com:mongodb-js/vscode into gagik/filter…
gagik Nov 11, 2024
3c63e76
Use helpers for tests
gagik Nov 12, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 37 additions & 5 deletions src/participant/prompts/promptBase.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ export interface PromptArgsBase {
};
context?: vscode.ChatContext;
connectionNames?: string[];
databaseName?: string;
collectionName?: string;
}

export interface UserPromptResponse {
Expand Down Expand Up @@ -163,16 +165,27 @@ export abstract class PromptBase<TArgs extends PromptArgsBase> {
protected getHistoryMessages({
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This function is starting to get a bit long and hard to follow all of the things happening in it.
It already has the // eslint-disable-next-line complexity which is usually an indicator that we should break it into a few functions, even if it comes with a slightly hit to performance (like running through the messages multiple times).
Should we do that now? Break this function into multiple parts where each is doing a certain thing? That'll also make it more easily unit testable.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More like a question about this area in code and maybe a bit of a request :) if you will refactor this code, could we split the history into something like getUserHistoryMessages and getAssistantHistoryMessages. When testing this functionality I found it difficult to print the history content because it is already wrapped into the vscode.LanguageModelChatMessage format when it is returned here: https://github.com/mongodb-js/vscode/blob/main/src/participant/prompts/promptBase.ts#L130

Maybe, it could be something like:

const messages = [
  vscode.LanguageModelChatMessage.Assistant(this.getAssistantPrompt(args)),
  vscode.LanguageModelChatMessage.Assistant(this.getAssistantHistoryMessages()),
  vscode.LanguageModelChatMessage.User(this.getUserHistoryMessages()),
  vscode.LanguageModelChatMessage.User(prompt),
];

Or something like that. The idea here is that we have some unformatted string value to print to see what message we send to the model.

Copy link
Member

@Anemy Anemy Nov 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ordering of the messages here is important, although the last message will always be the user prompt and the first is the assistant prompt, the user history and assistant history aren't sequential.
We do currently log information about the messages we are sending to the model:

messages: modelInput.messages.map(

@alenakhineika we can add something there that will log them in their entirety, and not just the metadata, if an environment variable is set. How does that sound?

Copy link
Contributor Author

@gagik gagik Nov 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am also dealing changing getHistoryMessages in VSCODE-632 so I think it'll be best to deal with some refactoring for it in the PR there.

connectionNames,
context,
databaseName,
collectionName,
}: {
connectionNames?: string[]; // Used to scrape the connecting messages from the history.
context?: vscode.ChatContext;
databaseName?: string;
collectionName?: string;
}): vscode.LanguageModelChatMessage[] {
const messages: vscode.LanguageModelChatMessage[] = [];

if (!context) {
return [];
}

let previousItem:
| vscode.ChatRequestTurn
| vscode.ChatResponseTurn
| undefined = undefined;

const namespaceIsKnown =
databaseName !== undefined && collectionName !== undefined;
for (const historyItem of context.history) {
if (historyItem instanceof vscode.ChatRequestTurn) {
if (
Expand All @@ -181,9 +194,21 @@ export abstract class PromptBase<TArgs extends PromptArgsBase> {
) {
// When the message is empty or a connection name then we skip it.
// It's probably going to be the response to the connect step.
previousItem = historyItem;
continue;
}

if (previousItem instanceof vscode.ChatResponseTurn) {
const responseIntent = (previousItem.result as ChatResult).metadata
?.intent;

// If the namespace is already known, skip responses to prompts asking for it.
if (responseIntent === 'askForNamespace' && namespaceIsKnown) {
previousItem = historyItem;
continue;
}
}

// eslint-disable-next-line new-cap
messages.push(vscode.LanguageModelChatMessage.User(historyItem.prompt));
}
Expand All @@ -206,11 +231,17 @@ export abstract class PromptBase<TArgs extends PromptArgsBase> {
'emptyRequest',
'askToConnect',
];
if (
responseTypesToSkip.indexOf(
(historyItem.result as ChatResult)?.metadata?.intent
) > -1
) {

const responseType = (historyItem.result as ChatResult)?.metadata
?.intent;
if (responseTypesToSkip.includes(responseType)) {
previousItem = historyItem;
continue;
}

// If the namespace is already known, skip including prompts asking for it.
if (responseType === 'askForNamespace' && namespaceIsKnown) {
previousItem = historyItem;
continue;
}

Expand All @@ -232,6 +263,7 @@ export abstract class PromptBase<TArgs extends PromptArgsBase> {
// eslint-disable-next-line new-cap
messages.push(vscode.LanguageModelChatMessage.Assistant(message));
}
previousItem = historyItem;
}

return messages;
Expand Down
Loading
Loading