Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(node): Add @vercel/ai instrumentation #13892

Merged
merged 11 commits into from
Dec 9, 2024
Merged

feat(node): Add @vercel/ai instrumentation #13892

merged 11 commits into from
Dec 9, 2024

Conversation

AbhiPrasad
Copy link
Member

@AbhiPrasad AbhiPrasad commented Oct 7, 2024

Adds Sentry tracing instrumentation for the ai library.

For more information, see the ai documentation.

 const Sentry = require('@sentry/node');

 Sentry.init({
  integrations: [Sentry.vercelAIIntegration()],
 });

By default this integration adds tracing support to all ai callsites. If you need to disable
collecting spans for a specific call, you can do so by setting experimental_telemetry.isEnabled to
false in the first argument of the function call.

const result = await generateText({
  model: openai('gpt-4-turbo'),
  experimental_telemetry: { isEnabled: false },
});

If you want to collect inputs and outputs for a specific call, you must specifically opt-in to each
function call by setting experimental_telemetry.recordInputs and experimental_telemetry.recordOutputs
to true.

const result = await generateText({
 model: openai('gpt-4-turbo'),
 experimental_telemetry: { isEnabled: true, recordInputs: true, recordOutputs: true },
});

resolves #13679

@AbhiPrasad AbhiPrasad self-assigned this Oct 7, 2024
Copy link
Contributor

github-actions bot commented Oct 7, 2024

size-limit report 📦

Path Size % Change Change
@sentry/browser 23.19 KB - -
@sentry/browser - with treeshaking flags 21.87 KB - -
@sentry/browser (incl. Tracing) 35.68 KB - -
@sentry/browser (incl. Tracing, Replay) 72.91 KB - -
@sentry/browser (incl. Tracing, Replay) - with treeshaking flags 63.31 KB - -
@sentry/browser (incl. Tracing, Replay with Canvas) 77.22 KB - -
@sentry/browser (incl. Tracing, Replay, Feedback) 89.71 KB - -
@sentry/browser (incl. Feedback) 39.95 KB - -
@sentry/browser (incl. sendFeedback) 27.81 KB - -
@sentry/browser (incl. FeedbackAsync) 32.6 KB - -
@sentry/react 25.87 KB - -
@sentry/react (incl. Tracing) 38.5 KB - -
@sentry/vue 27.4 KB - -
@sentry/vue (incl. Tracing) 37.52 KB - -
@sentry/svelte 23.36 KB - -
CDN Bundle 24.36 KB - -
CDN Bundle (incl. Tracing) 37.34 KB - -
CDN Bundle (incl. Tracing, Replay) 72.53 KB - -
CDN Bundle (incl. Tracing, Replay, Feedback) 77.89 KB - -
CDN Bundle - uncompressed 71.56 KB - -
CDN Bundle (incl. Tracing) - uncompressed 110.85 KB - -
CDN Bundle (incl. Tracing, Replay) - uncompressed 224.92 KB - -
CDN Bundle (incl. Tracing, Replay, Feedback) - uncompressed 238.13 KB - -
@sentry/nextjs (client) 38.79 KB - -
@sentry/sveltekit (client) 36.18 KB - -
@sentry/node 163.66 KB +0.45% +745 B 🔺
@sentry/node - without tracing 99.81 KB - -
@sentry/aws-serverless 127.3 KB - -

View base workflow run

@AbhiPrasad AbhiPrasad requested review from a team, Lms24 and s1gr1d and removed request for a team November 27, 2024 13:28
@AbhiPrasad AbhiPrasad marked this pull request as ready for review November 27, 2024 13:28
@AbhiPrasad AbhiPrasad requested a review from mydea November 27, 2024 13:28
Copy link

codecov bot commented Nov 27, 2024

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
683 1 682 299
View the top 1 failed tests by shortest run time
client-app-routing-instrumentation.test.ts Creates a navigation transaction for app router routes
Stack Traces | 30s run time
client-app-routing-instrumentation.test.ts:19:5 Creates a navigation transaction for app router routes

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

Copy link
Member

@s1gr1d s1gr1d left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me :)

Comment on lines +161 to +166
* Adds Sentry tracing instrumentation for the [ai](https://www.npmjs.com/package/ai) library.
*
* For more information, see the [`ai` documentation](https://sdk.vercel.ai/docs/ai-sdk-core/telemetry).
*
* @example
* ```javascript
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice JS doc 👍

Copy link
Member

@Lms24 Lms24 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just once concern, otherwise LGTM!

attributes['ai.usage.completionTokens'] != undefined &&
attributes['ai.usage.promptTokens'] != undefined
) {
span.data['ai.tokens.used'] = attributes['ai.usage.completionTokens'] + attributes['ai.usage.promptTokens'];
Copy link
Member

@Lms24 Lms24 Dec 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking at our LLM monitoring attribute spec, should this rather be:

Suggested change
span.data['ai.tokens.used'] = attributes['ai.usage.completionTokens'] + attributes['ai.usage.promptTokens'];
span.data['ai.total_tokens.used'] = attributes['ai.usage.completionTokens'] + attributes['ai.usage.promptTokens'];

? Not sure though, maybe there's another spec somewhere for this attribute?

@AbhiPrasad AbhiPrasad enabled auto-merge (squash) December 5, 2024 23:08
@AbhiPrasad AbhiPrasad merged commit 08b7e49 into develop Dec 9, 2024
156 checks passed
@AbhiPrasad AbhiPrasad deleted the abhi-vercel-ai branch December 9, 2024 16:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add instrumentation for ai
3 participants