refactor: add telemetry for office agent #11831

knightlea · 2024-06-14T07:47:29Z

Add telemetry for office agent
PM spec: https://microsoft.sharepoint.com/:w:/t/OfficePlatform/ESZVb6Q77nZBhXZr1pWEVaABmLATm2tHDMEPkuo7ITgVvg?e=0LwZO8

GavinGu07 · 2024-06-17T03:02:35Z

packages/vscode-extension/src/officeChat/utils.ts

  const purifiedResult = await getCopilotResponseAsString(
    "copilot-gpt-4",
    purifyUserMessage,
    token
  );
+  const t1 = performance.now();
+  const requestTokens = countMessagesTokens(purifyUserMessage);
+  const responseTokens = Tokenizer.getInstance().tokenLength(purifiedResult);


Use countMessagesTokens here. Same for the rest.

done, thanks

… user/haojiangliu/addTelemetry

packages/vscode-extension/src/officeChat/commands/create/helper.ts

packages/vscode-extension/src/officeChat/common/planner.ts

packages/vscode-extension/src/officeChat/common/samples/sampleProvider.ts

packages/vscode-extension/src/officeChat/commands/nextStep/officeNextstepCommandHandler.ts

packages/vscode-extension/src/officeChat/common/samples/sampleProvider.ts

packages/vscode-extension/src/officeChat/handlers.ts

codecov · 2024-06-24T11:02:07Z

Codecov Report

Attention: Patch coverage is 90.00000% with 21 lines in your changes missing coverage. Please review.

Project coverage is 89.40%. Comparing base (557dab3) to head (e2ccfc1).
Report is 39 commits behind head on dev.

Additional details and impacted files

@@            Coverage Diff             @@
##              dev   #11831      +/-   ##
==========================================
+ Coverage   89.34%   89.40%   +0.05%     
==========================================
  Files         566      569       +3     
  Lines       33150    33311     +161     
  Branches     6417     6541     +124     
==========================================
+ Hits        29619    29781     +162     
+ Misses       1612     1601      -11     
- Partials     1919     1929      +10

Files	Coverage Δ
...extension/src/officeChat/commands/create/helper.ts	`99.09% <100.00%> (+0.02%)`	⬆️
.../commands/nextStep/officeNextstepCommandHandler.ts	`96.61% <100.00%> (+0.31%)`	⬆️
...on/src/officeChat/common/samples/sampleProvider.ts	`92.63% <100.00%> (+0.18%)`	⬆️
...sion/src/officeChat/common/skills/codeExplainer.ts	`100.00% <100.00%> (ø)`
...src/officeChat/common/skills/codeIssueCorrector.ts	`96.40% <100.00%> (+0.05%)`	⬆️
...-extension/src/officeChat/common/skills/printer.ts	`100.00% <100.00%> (ø)`
...ion/src/officeChat/common/skills/projectCreator.ts	`100.00% <ø> (ø)`
...ode-extension/src/officeChat/common/skills/spec.ts	`100.00% <100.00%> (ø)`
packages/vscode-extension/src/officeChat/utils.ts	`100.00% <100.00%> (ø)`
...code-extension/src/telemetry/extTelemetryEvents.ts	`100.00% <100.00%> (ø)`
... and 6 more

... and 11 files with indirect coverage changes

GavinGu07 · 2024-06-25T09:34:21Z

packages/vscode-extension/src/officeChat/telemetry.ts

+  OffTopic = "Off Topic",
+  LanguageModelError = "LanguageModel Error",
+}
+export class OfficeChatTelemetryData extends ChatTelemetryData {


If you are implementing all member variables/methods, please implement the interface IChatTelemetryData instead of extend ChatTelemetryData

will fix, thanks

GavinGu07 · 2024-06-25T11:29:31Z

packages/vscode-extension/src/officeChat/commands/create/helper.ts

@@ -95,11 +112,11 @@ export function getOfficeTemplateMetadata(): ProjectMetadata[] {
 export async function showOfficeSampleFileTree(
  projectMetadata: ProjectMetadata,
  response: ChatResponseStream
-): Promise<string> {
+): Promise<object> {


Define a object struct { path: string, host: string } instead of returning an object

will fix, thanks

GavinGu07 · 2024-06-25T11:33:48Z

packages/vscode-extension/src/officeChat/commands/create/officeCreateCommandHandler.ts

+        const sampleInfos = await showOfficeSampleFileTree(matchedResult, response);
+        const folder = (sampleInfos as any)?.["path"];
+        const hostType = (sampleInfos as any)?.["host"].toLowerCase();


Use the defined object structure to fetch these variables instead of cast to any

will fix, thanks

GavinGu07 · 2024-06-25T11:35:38Z

packages/vscode-extension/src/officeChat/commands/create/officeCreateCommandHandler.ts

        const sampleTitle = localize("teamstoolkit.chatParticipants.create.sample");
+        officeChatTelemetryData.setHostType(hostType);
+        const matchResultInfo = "sample";


This has been defined in matchedResult.type, remove this variable.

will fix, thanks

GavinGu07 · 2024-06-25T11:36:47Z

packages/vscode-extension/src/officeChat/commands/create/officeCreateCommandHandler.ts

+        const tmpHostType = (matchedResult.data as any)?.["addin-host"].toLowerCase();
+        const tmpFolder = await showOfficeTemplateFileTree(matchedResult.data as any, response);


Same as above, use the defined object struct.

the type of matchedResult.data is unknown so that I think it should be convert to any

GavinGu07 · 2024-06-25T11:37:20Z

packages/vscode-extension/src/officeChat/commands/create/officeCreateCommandHandler.ts

        const templateTitle = localize("teamstoolkit.chatParticipants.create.template");
+        officeChatTelemetryData.setHostType(tmpHostType);
+        const tmpmatchResultInfo = "template";


Remove and use matchedResult.type

will fix, thanks

GavinGu07 · 2024-06-25T12:03:25Z

packages/vscode-extension/src/officeChat/common/planner.ts

    response.markdown(`
 ${localize("teamstoolkit.chatParticipants.officeAddIn.printer.outputTemplate.intro")}\n
 ${purified}
 `);
    const spec = new Spec(purified);
+    spec.appendix.telemetryData.requestId = telemetryData.requestId;


Put it into the constructor

will fix, thanks

GavinGu07 · 2024-06-25T12:05:37Z

packages/vscode-extension/src/officeChat/common/planner.ts

+    // telemetryData.setCodeClassAndMembers(
+    //   spec.appendix.telemetryData.codeClassAndMembers.toString()
+    // );


Remove these lines

will remove, thanks

GavinGu07 · 2024-06-25T12:33:48Z

packages/vscode-extension/src/officeChat/commands/create/helper.ts

+  const t1 = performance.now();
+  telemetryData.responseTokensPerRequest.push(
+    OfficeChatTelemetryData.calculateResponseTokensPerRequest(response, t0, t1)
+  );
+  telemetryData.chatMessages.push(
+    ...messages,
+    new LanguageModelChatMessage(LanguageModelChatMessageRole.Assistant, response)
+  );


Maybe log the response tokens/LLM call is unnecessary. We can only log all response tokens in an array and calculate this metrics in markComplete. Please feel free to call out if you have other thoughts.

have fixed to distinguish request token and response token and removed response tokens per request

GavinGu07 · 2024-06-26T08:11:36Z

packages/vscode-extension/src/officeChat/telemetry.ts

+      // this.telemetryData.properties[TelemetryProperty.CopilotChatCodeClassAndMembers] =
+      //   this.codeClassAndMembers;


Remove these lines

will remove, thanks

GavinGu07 · 2024-06-26T08:11:56Z

packages/vscode-extension/src/telemetry/extTelemetryEvents.ts

+  CopilotMatchResultType = "copilot-match-result-type",
+  CopilotChatBlockReason = "copilot-chat-block-reason",
+  CopilotChatRelatedSampleName = "copilot-chat-related-sample-name",
+  CopilotChatCodeClassAndMembers = "copilot-chat-code-class-and-members",


Remove this unused variable

done, thanks

GavinGu07 · 2024-06-26T08:13:11Z

packages/vscode-extension/src/telemetry/extTelemetryEvents.ts

+  CopilotChatTimeToFirstToken = "copilot-chat-time-to-first-token",
+  CopilotChatTotalTokensPerSecond = "copilot-chat-total-tokens-per-second",
+  CopilotChatResponseTokensPerRequest = "copilot-chat-response-tokens-per-request",
+  CopilotChatTotalTokens = "copilot-chat-total-tokens",


There is existing CopilotChatTokenCount, so we should just use that.

the existing name could not distinguish request token and response token. I have added 2 new telemetry names to make it clear

…elemetry

packages/vscode-extension/src/officeChat/commands/create/officeCreateCommandHandler.ts

packages/vscode-extension/src/officeChat/common/skills/codeExplainer.ts

packages/vscode-extension/src/officeChat/common/samples/sampleProvider.ts

packages/vscode-extension/src/officeChat/common/skills/codeIssueCorrector.ts

packages/vscode-extension/src/officeChat/common/skills/codeGenerator.ts

packages/vscode-extension/src/officeChat/telemetry.ts

GavinGu07 · 2024-06-27T02:50:51Z

packages/vscode-extension/src/officeChat/utils.ts

@@ -102,5 +123,5 @@ export async function getOfficeSampleDownloadUrlInfo(sampleId: string) {
  if (!sample) {
    throw new Error("Sample not found");
  }
-  return sample.downloadUrlInfo;
+  return { downloadUrlInfo: sample.downloadUrlInfo, host: sample.types[0] };


Change the function name to getOfficeSample and return the sample object in this method. Fetch the downloadUrlInfo and host type in outer calling function.

done, thanks

GavinGu07 · 2024-06-27T02:54:04Z

packages/vscode-extension/src/officeChat/common/skills/codeGenerator.ts

+    //       groupedMethodsOrProperties.forEach((methodsOrPropertiesCandidates, className) => {
+    //         for (let i = 0; i < methodsOrPropertiesCandidates.length; i++) {
+    //           let methodOrProperty = methodsOrPropertiesCandidates[i].codeSample;
+    //           if (methodOrProperty.startsWith("readonly ")) {
+    //             methodOrProperty = methodOrProperty.replace("readonly ", "");
+    //           }
+    //           const lastColonIndex = methodOrProperty.lastIndexOf(":");
+    //           if (lastColonIndex !== -1) {
+    //             methodOrProperty = methodOrProperty.substring(0, lastColonIndex);
+    //           }
+    //           const classCode = `${className}.${methodOrProperty}`;
+    //           spec.appendix.telemetryData.codeClassAndMembers.push(classCode);
+    //         }
+    //       });


Remove these lines if never used.

GavinGu07 · 2024-06-27T02:54:16Z

packages/vscode-extension/src/officeChat/common/skills/codeGenerator.ts

@@ -367,6 +383,7 @@ ${spec.appendix.codeExplanation
    //         host,
    //         codeSpec,
    //         "" //sampleCode
+    //         spec


Remove this line if never used.

I have also changed the function getMostRelevantDeclarationsUsingLLM because there is an interaction with copilot in it. seems like there is no other usage of the function in other code. should I also remove my change on the function?

GavinGu07 · 2024-06-27T02:57:32Z

packages/vscode-extension/src/officeChat/commands/create/officeCreateCommandHandler.ts

@@ -40,8 +41,8 @@ export default async function officeCreateCommandHandler(

  if (request.prompt.trim() === "") {
    response.markdown(localize("teamstoolkit.chatParticipants.officeAddIn.create.noPromptAnswer"));
-
-    officeChatTelemetryData.markComplete();
+    officeChatTelemetryData.setBlockReason("Empty Input");


Should the 'Empty Input' be set in an Enum?

yes I missed it. will fix, thanks

knightlea · 2024-06-27T06:22:18Z

packages/vscode-extension/src/officeChat/types.ts

@@ -11,3 +11,8 @@ export interface ICopilotChatOfficeResultMetadata {
 export interface ICopilotChatOfficeResult extends ChatResult {
  readonly metadata?: ICopilotChatOfficeResultMetadata;
 }
+
+export type ProjectMiniData = {


Sure, thanks

GavinGu07 · 2024-06-27T06:22:51Z

packages/vscode-extension/src/officeChat/commands/create/helper.ts

  response.markdown(
    "\nWe've found a sample project that matches your description. Take a look at it below."
  );
-  const downloadUrlInfo = await getOfficeSampleDownloadUrlInfo(projectMetadata.id);
+  const { downloadUrlInfo, host } = await getOfficeSampleDownloadUrlInfo(projectMetadata.id);


How about the host type of the matched templates? seems like there's only telemetry for matched samples.

it has been already logged. thanks

GavinGu07 · 2024-07-01T07:01:16Z

packages/vscode-extension/src/officeChat/common/planner.ts

+          if (spec.appendix.telemetryData.isHarmful === false) {
+            telemetryData.setBlockReason(OfficeChatTelemetryBlockReasonEnum.RAI);
+          }


Should it be set when isHamrful === true?

packages/vscode-extension/src/officeChat/commands/create/helper.ts

refactor: add telemetry for office agent

fbdeeac

knightlea requested review from 1openwindow, HuihuiWu-Microsoft, nliu-ms and tecton as code owners June 14, 2024 07:47

GavinGu07 reviewed Jun 17, 2024

View reviewed changes

theRealHaojiangLiu added 2 commits June 19, 2024 12:45

Merge branch 'dev' of https://github.com/OfficeDev/teams-toolkit into…

8a9e0b9

… user/haojiangliu/addTelemetry

fix: add telemetry for office agent

58626d5

github-advanced-security bot found potential problems Jun 19, 2024

View reviewed changes

packages/vscode-extension/src/officeChat/commands/create/helper.ts Fixed Show fixed Hide fixed

packages/vscode-extension/src/officeChat/common/planner.ts Fixed Show resolved Hide resolved

packages/vscode-extension/src/officeChat/common/samples/sampleProvider.ts Fixed Show fixed Hide fixed

fix: add telemetry for office agent for unit test

b021495

github-advanced-security bot found potential problems Jun 19, 2024

View reviewed changes

theRealHaojiangLiu added 6 commits June 19, 2024 17:20

fix: add telemetry for office agent for unit test 2

97f3375

fix: add telemetry for office agent for unit test - bug fix

b45eae3

fix: add telemetry for office agent for unit test - bug fix 2

7e4566a

fix: add telemetry for office agent for unit test - bug fix 3

b7f16c9

fix: add telemetry for office agent for unit test - remove no use import

773470c

fix: add telemetry for office agent for unit test - resolve confilt

4d39e0c

GavinGu07 reviewed Jun 25, 2024

View reviewed changes

GavinGu07 reviewed Jun 26, 2024

View reviewed changes

theRealHaojiangLiu added 2 commits June 26, 2024 18:09

Merge remote-tracking branch 'teamsfx/dev' into user/haojiangliu/addT…

d1d0637

…elemetry

fix: add telemetry for office agent for unit test - resolve comments

3b6a02a

github-advanced-security bot found potential problems Jun 26, 2024

View reviewed changes

GavinGu07 reviewed Jun 27, 2024

View reviewed changes

knightlea commented Jun 27, 2024

View reviewed changes

GavinGu07 reviewed Jun 27, 2024

View reviewed changes

theRealHaojiangLiu added 7 commits June 27, 2024 15:26

fix: add telemetry for office agent for unit test - resolve comments

fd54ca1

fix: add telemetry for office agent for unit test - bug fix 4

f4a5120

fix: add telemetry for office agent for unit test - bug fix 5

3805593

fix: add telemetry for office agent for unit test - bug fix 6

1b7b992

fix: add telemetry for office agent for unit test - bug fix 7

59afe81

fix: add telemetry for office agent for unit test - bug fix

e50790b

fix: add telemetry for office agent for unit test - bug fix 9

1a49d3b

GavinGu07 reviewed Jul 1, 2024

View reviewed changes

theRealHaojiangLiu added 3 commits July 1, 2024 15:59

fix: add telemetry for office agent for unit test - bug fix 10

b7807f7

fix: add telemetry for office agent for unit test - bug fix 11

f9e27a1

fix: add telemetry for office agent for unit test - bug fix 12

20255c3

github-advanced-security bot found potential problems Jul 1, 2024

View reviewed changes

packages/vscode-extension/src/officeChat/commands/create/helper.ts Fixed Show resolved Hide resolved

fix: add telemetry for office agent for unit test - bug fix 13

e2ccfc1

GavinGu07 approved these changes Jul 2, 2024

View reviewed changes

1openwindow approved these changes Jul 2, 2024

View reviewed changes

1openwindow merged commit 88458b9 into OfficeDev:dev Jul 2, 2024
18 checks passed

		const tmpHostType = (matchedResult.data as any)?.["addin-host"].toLowerCase();
		const tmpFolder = await showOfficeTemplateFileTree(matchedResult.data as any, response);

		// this.telemetryData.properties[TelemetryProperty.CopilotChatCodeClassAndMembers] =
		// this.codeClassAndMembers;

refactor: add telemetry for office agent #11831

refactor: add telemetry for office agent #11831

Conversation

knightlea commented Jun 14, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 24, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GavinGu07 Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

knightlea Jun 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

knightlea Jun 27, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Jun 24, 2024 •

edited

Loading

GavinGu07 Jun 26, 2024 •

edited

Loading

knightlea Jun 27, 2024 •

edited

Loading

knightlea Jun 27, 2024 •

edited

Loading