feat(participant): route generic prompt by intent and update generic prompt VSCODE-572 #830
Conversation
| - Default | ||
| Rules: | ||
| 1. Respond only with the intent handler. | ||
| 2. Use the "Query" intent handler when the user is asking for code that relates to a specific collection. |
There was a problem hiding this comment.
What is the user has not specified a namespace? We don't have information about a specific collection, but it still can be a query generation request.
There was a problem hiding this comment.
That's a good question, and raises a tradeoff we have to take. Some users may ask a question that would result in code for a database, like how can I create a user with specific database permissions or what's the size of X database. In that case the /query handler would be forcing them to choose a collection which doesn't really make sense. I was aiming to make this prompt fairly intentionally selective so we don't restrict the user capabilities. The generic prompt has a lot of the capabilities of the /query resolver.
Are there ways you think we could word this differently or guide the behavior to the query resolver better?
| Rules: | ||
| 1. Respond only with the intent handler. | ||
| 2. Use the "Query" intent handler when the user is asking for code that relates to a specific collection. | ||
| 3. Use the "Docs" intent handler when the user is asking a question that involves MongoDB documentation. |
There was a problem hiding this comment.
Did you try this out? I am wondering how the model interprets this instruction. Should users explicitly say find in the documentation? Or they can just ask "How to..." questions?
There was a problem hiding this comment.
The model did successfully route a fairly docs related question to the docs handler in my manual testing. Haven't been that extensive to give a good answer on when it does and when it doesn't. We'd need more accuracy tests.
| 2. You should suggest code that is performant and correct. | ||
| 3. Respond with markdown. | ||
| 4. When relevant, provide code in a Markdown code block that begins with \`\`\`javascript and ends with \`\`\`. | ||
| 5. Use MongoDB shell syntax for code unless the user requests a specific language. |
There was a problem hiding this comment.
This is just a reminder that code blocks should be aligned with codeBlockIdentifier from #835
| @@ -34,7 +34,7 @@ type AssertProps = { | |||
|
|
|||
| type TestCase = { | |||
| testCase: string; | |||
| type: 'generic' | 'query' | 'namespace'; | |||
| type: 'intent' | 'generic' | 'query' | 'namespace'; | |||
There was a problem hiding this comment.
Is there a docs type?
There was a problem hiding this comment.
This is for the accuracy tests, we don't have any docs tests yet. Once we do we can add it.
* VSCODE-528-mongodb-copilot: fix(chat): update response handling to stream and inline code block parsing VSCODE-620 (#835) feat(participant): route generic prompt by intent and update generic prompt VSCODE-572 (#830) # Conflicts: # src/participant/participant.ts # src/participant/prompts/query.ts
VSCODE-572
This pr adds an
intentprompt. This is used when a user adds a message without a command handler. It tries to route the user's message to a command handler if possible, and if it doesn't route it'll go to the default/generic handler.There are a few accuracy test adds as well, as well as adding links to the test names for quicker reference in the html.
