Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(chat): Report docs fallback telemetry #825

Merged
merged 1 commit into from
Sep 23, 2024
Merged

feat(chat): Report docs fallback telemetry #825

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions src/participant/participant.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1120,6 +1120,14 @@ export default class ParticipantController {
// If the docs chatbot API is not available, fall back to Copilot’s LLM and include
// the MongoDB documentation link for users to go to our documentation site directly.
log.error(error);
this._telemetryService.track(
TelemetryEventTypes.PARTICIPANT_RESPONSE_FAILED,
{
command: 'docs',
error_name: ParticipantErrorTypes.DOCS_CHATBOT_API,
}
);

docsResult = await this._handleDocsRequestWithCopilot(...args);
}

Expand Down
3 changes: 2 additions & 1 deletion src/telemetry/telemetryService.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ type ParticipantFeedbackProperties = {
type ParticipantResponseFailedProperties = {
command: string;
error_code?: string;
error_name: string;
error_name: ParticipantErrorTypes;
};

export function chatResultFeedbackKindToTelemetryValue(
Expand Down Expand Up @@ -167,6 +167,7 @@ export enum ParticipantErrorTypes {
INVALID_PROMPT = 'Invalid Prompt',
FILTERED = 'Filtered by Responsible AI Service',
OTHER = 'Other',
DOCS_CHATBOT_API = 'Docs Chatbot API Issue',
}

/**
Expand Down
21 changes: 12 additions & 9 deletions src/test/suite/participant/participant.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@ suite('Participant Controller Test Suite', function () {
};
let chatTokenStub;
let countTokensStub;
let sendRequestStub;
let sendRequestStub: sinon.SinonStub;
let telemetryTrackStub: SinonSpy;

const invokeChatHandler = async (
Expand Down Expand Up @@ -399,13 +399,6 @@ suite('Participant Controller Test Suite', function () {
TelemetryEventTypes.PARTICIPANT_WELCOME_SHOWN
);
expect(telemetryTrackStub.lastCall.args[1]).to.be.undefined;

telemetryTrackStub
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This must have been a bad merge that I didn't notice.

.getCalls()
.map((call) => call.args[0])
.filter(
(arg) => arg === TelemetryEventTypes.PARTICIPANT_WELCOME_SHOWN
).length;
});
});

Expand Down Expand Up @@ -1226,7 +1219,7 @@ Schema:

suite('docs command', function () {
const initialFetch = global.fetch;
let fetchStub;
let fetchStub: sinon.SinonStub;

beforeEach(function () {
sendRequestStub.onCall(0).resolves({
Expand Down Expand Up @@ -1275,6 +1268,16 @@ Schema:
};
await invokeChatHandler(chatRequestMock);
expect(sendRequestStub).to.have.been.called;

// Expect the error to be reported through the telemetry service
sinon.assert.calledOnce(telemetryTrackStub);
expect(telemetryTrackStub.lastCall.args[0]).to.equal(
TelemetryEventTypes.PARTICIPANT_RESPONSE_FAILED
);

const properties = telemetryTrackStub.lastCall.args[1];
expect(properties.command).to.equal('docs');
expect(properties.error_name).to.equal('Docs Chatbot API Issue');
});
});
});
Expand Down
Loading