feat(audoedit): implement basic analytics logger

[valerybugakov]

• The AutoeditAnalyticsLogger is an improved version of the autocomplete analytics logger without persistence tracking. It's much more type-safe than its predecessor, with explicitly defined state transitions and required attributes for each stage. Plus, many unused attributes and logic were removed.
• At the same time, we want to preserve all the small details that helped us to collect valuable data and filter out the noise. If you see anything important that's missing from the autocomplete logger, please call it out!
• Closes CODY-4557: Telemetry: implement basic autoedits logger

Test plan

• CI + unit tests
• The integration with the actual autoedits provider will implemented in a follow-up PR.

[hitesh-1997]

How can we transition from suggestion to accepted or rejected without read ?
If we think there is an intermediate stage where although we suggest a prediction but maybe user typed something so that it was not read, do you think we should introduce another state something like noRead so that suggested can have transition to either read or noRead and noRead would have empty transition ?

[valerybugakov]

We consider suggestions as read after a hard-coded timeout if they were not rejected during this time. Upon reviewing the code, I see that we don't need this data to send the existing event payloads. We use displayDuration, which does not require readAt to be computed.

I'll remove this logic altogether. We can add it later if we think we need it.

[hitesh-1997]

We typically use lineCount and charCount to calculate stats over newly added code in the editor. But for the auto-edits the values typically include existing code as well, so we may need to adjust the metrics accordingly.

[valerybugakov]

Agree, let's adjust in a follow-up PR.

[hitesh-1997]

Another field which would be helpful here is:
is code_to_rewrite same as prediction. Since auto-edits trigger suggestion on cursor movement, a high value suggestion could indicate irrelevant suggestion.

[valerybugakov]

Let's add this in a follow-up PR. We might want an enum field indicating the reason for a suggestion to be hidden.

[hitesh-1997]

what cases does no response happen ? Is this logged when the response is empty string from the model ? If yes, I would suspect this to not trigger since unlike autocomplete, the model atleast rewrite the codeToRewrite and empty response here means that we delete all the code in the codeToRewrite section.

[valerybugakov]

In autocomplete, this happens when we hide a suggestion for whatever reason, such as when the suggestion duplicates the document suffix. We can extend this even with extra fields in follow-up PRs.

[hitesh-1997]

gotcha !!

[valerybugakov]

I changed this event name to discarded and added some comments to clarify it. We can adjust it follow-ups based on use cases.

[hitesh-1997]

For all the states do you think we should just have

1. phase
2. payload
   All the payload at any state anyways extend the previous state. So we can move the specific fields like startedAt, loadedAt etc in the metadata itself. So, that we don't have to extend both the metadata and the states.

So, after this the metadata would be incorporate additional ields like loadedAt, readAt etc and each state would have no need to extend previous state and Omit some fields from the previous state.

Currently, introducing any new start in between requires this duplication of extending metadata/state fields.

[valerybugakov]

Yeah, this structure requires a bit of extra code. However, based on our experience with the autocomplete analytics logger, it's necessary always to have a type-safe setup for the fields we send to our analytics backend.

I updated a top-level comment explaining the rationale here. Here are the two most relevant points:

 * 3. The `payload` field in each state encapsulates the exact list of fields that we plan to send
 *    to our analytics backend.
 *
 * 4. Other top-level `state` fields are saved only for bookkeeping and won't end up at our
 *    analytics backend. This ensures we don't send unintentional or redundant information to
 *    the analytics backend.

That's why I'd like to separate payload and other bookkeeping fields (startedAt, etc).

[hitesh-1997]

For the privateMetadata should we add recordsPrivateMetadataTranscript: 1 in the metadata ?
Also adding @akalia25 for a look here

[valerybugakov]

As per the docs, it's required if we want to send promptText or responseText, which we don't have in this logger yet. Should we add recordsPrivateMetadataTranscript later when we need it?

[hitesh-1997]

I noticed earlier that we log prediction?: string field which would be responseText so suggested the change. yes, we can add it later when we log responseText

[valerybugakov]

Ok, I reread this Slack message and agree that we should add it here.

I had an impression that it's required for long string values, so I wasn't concerned about the prediction field because it's limited to 300 chars, but it looks like we have to add it because autoedit event may contain sensitive data

[valerybugakov]

It probably means we should add it to autocomplete events containing the insertText field too.

[valerybugakov]

Added here: 3570628

[akalia25]

Thanks for the tag, @hitesh-1997!

Yes, when recording sensitive data (e.g., transcripts, or free-form text), we should always add recordsPrivateMetadataTranscript: 1 to the metadata. This helps us keep track of where sensitive data is being logged and route the data correctly. More details here

We need to know which key within privateMetadata contains the sensitive data, so if the key is not in the list below (i.e. insertText), then you'd simply need to message the discuss-analytics channel and we'd be happy to add the key.

promptText
responseText
inlineCompletionItemContext
privateContextSummary
diff
query

[hitesh-1997]

Awesome 🚀🚀

[valerybugakov]

Hey @kelsey-brown, @akalia25, could you take a look? The list of currently implemented event names:

type AutoeditEventAction =
    | 'suggested'
    | 'accepted'
    | 'noResponse'
    | 'error'
    | `invalidTransitionTo${Capitalize<Phase>}`

[kelsey-brown]

These should be categorized as follows:

• suggested: billable
• accepted: core
• noResponse: billable
• error: null (uncategorized)
• invalid: null (uncategorized)

Thanks for checking!

[hitesh-1997]

Thanks for addressing the comments !!

[valerybugakov]

Enabling auto-merge since the logger isn't integrated with the feature yet, and the rollout is still limited to S2 users only. @kelsey-brown and @akalia25, feel free to drop any review comments; I’ll handle them in follow-up PRs.

[kelsey-brown]

@valerybugakov just to clarify, the event names here aren’t changing - just the way they’re logged technically, and some of the metadata? And similarly, we won’t still log events the old way in addition to this - this will be a full replacement?

If you see anything important that's missing from the autocomplete logger, please call it out!

It's a bit hard for me to tell which metadata fields have been removed based on the code here - there are a lot 😅 Are these events live on S2? If so, I could fire some events there and compare the payloads side by side to what we're getting from other instances.

[valerybugakov]

@valerybugakov just to clarify, the event names here aren’t changing - just the way they’re logged technically

@kelsey-brown, I'll share more information in a Slack thread today.