-
-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(node): Ensure request bodies are reliably captured for http requests #13746
Conversation
size-limit report 📦
|
❌ 1 Tests Failed:
View the top 1 failed tests by shortest run time
To view individual test run time comparison to the main branch, go to the Test Analytics Dashboard |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Really cool change. I think we can clean up the flow a bit and I also hope that the patching doesn't blow up in any unexpected ways.
// This is non-standard, but may be set on e.g. Next.js or Express requests | ||
const cookies = (req as PolymorphicRequest).cookies; | ||
|
||
const normalizedRequest: Request = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
l/m: I have been burned in the past because Request
is a native type. I suggest we rename this to smth like NormalizedRequest
or ProcessingMetadataRequest
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this type already exists, it is not new 😬 but I do agree, but we probably need to alias it then and keep the old name around (deprecated)...?
@@ -148,7 +149,27 @@ export const instrumentHttp = Object.assign( | |||
const isolationScope = (scopes.isolationScope || getIsolationScope()).clone(); | |||
const scope = scopes.scope || getCurrentScope(); | |||
|
|||
const headers = req.headers; | |||
const host = headers.host || '<no host>'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
l: I haven't put too much thought into this but is <no host>
a good default?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yeah no idea, just kept this around ^^
@@ -101,7 +101,7 @@ export type ProfileItem = BaseEnvelopeItem<ProfileItemHeaders, Profile>; | |||
export type ProfileChunkItem = BaseEnvelopeItem<ProfileChunkItemHeaders, ProfileChunk>; | |||
export type SpanItem = BaseEnvelopeItem<SpanItemHeaders, Partial<SpanJSON>>; | |||
|
|||
export type EventEnvelopeHeaders = { event_id: string; sent_at: string; trace?: DynamicSamplingContext }; | |||
export type EventEnvelopeHeaders = { event_id: string; sent_at: string; trace?: Partial<DynamicSamplingContext> }; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
h: What is this about?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
sorry, forgot to mention this here: this was just not correctly annotated before, really. Since we did not type dynamicSamplingContext
on the sdkProcessingMetadata
before, this worked by accident. but because this is really Partial<DSC>
(based on what we set on it, which is also partial), it makes this necessary to be correct.
We could also choose to leave this and cast it in the place where we set this, but really that means lying to ourselves because we don't know it is complete. 🤷 no strong feelings though, if we feel this is a bad type change I can also cast it.
const { sdkProcessingMetadata = {} } = event; | ||
const req = sdkProcessingMetadata.request; | ||
const { request, normalizedRequest } = sdkProcessingMetadata; | ||
|
||
if (!req) { | ||
return event; | ||
const addRequestDataOptions = convertReqDataIntegrationOptsToAddReqDataOpts(_options); | ||
|
||
// If this is set, it takes precedence over the plain request object | ||
if (normalizedRequest) { | ||
// Some other data is not available in standard HTTP requests, but can sometimes be augmented by e.g. Express or Next.js | ||
const ipAddress = request ? request.ip || (request.socket && request.socket.remoteAddress) : undefined; | ||
const user = request ? request.user : undefined; | ||
|
||
return addNormalizedRequestDataToEvent(event, normalizedRequest, { ipAddress, user }, addRequestDataOptions); | ||
} | ||
|
||
const addRequestDataOptions = convertReqDataIntegrationOptsToAddReqDataOpts(_options); | ||
if (!request) { | ||
return event; | ||
} | ||
|
||
return addRequestDataToEvent(event, req, addRequestDataOptions); | ||
return addRequestDataToEvent(event, request, addRequestDataOptions); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would mayyyybee do it like this?:
const { sdkProcessingMetadata = {} } = event; | |
const req = sdkProcessingMetadata.request; | |
const { request, normalizedRequest } = sdkProcessingMetadata; | |
if (!req) { | |
return event; | |
const addRequestDataOptions = convertReqDataIntegrationOptsToAddReqDataOpts(_options); | |
// If this is set, it takes precedence over the plain request object | |
if (normalizedRequest) { | |
// Some other data is not available in standard HTTP requests, but can sometimes be augmented by e.g. Express or Next.js | |
const ipAddress = request ? request.ip || (request.socket && request.socket.remoteAddress) : undefined; | |
const user = request ? request.user : undefined; | |
return addNormalizedRequestDataToEvent(event, normalizedRequest, { ipAddress, user }, addRequestDataOptions); | |
} | |
const addRequestDataOptions = convertReqDataIntegrationOptsToAddReqDataOpts(_options); | |
if (!request) { | |
return event; | |
} | |
return addRequestDataToEvent(event, req, addRequestDataOptions); | |
return addRequestDataToEvent(event, request, addRequestDataOptions); | |
const { sdkProcessingMetadata = {} } = event; | |
const { request, normalizedRequest } = sdkProcessingMetadata; | |
const addRequestDataOptions = convertReqDataIntegrationOptsToAddReqDataOpts(_options); | |
if (request) { | |
addRequestDataToEvent(event, request, addRequestDataOptions); | |
} | |
// If this is set, it takes precedence over the plain request object | |
if (normalizedRequest) { | |
addNormalizedRequestDataToEvent(event, normalizedRequest, addRequestDataOptions); | |
} | |
return event; |
So that
- We always use all of the data.
- For now we can just use the old logic from
addRequestDataToEvent
to extract the "non-standard" fields on the request. Removing the need for theadditionalData
arg which I admittedly find a bit offputting.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Sorry, took some time to come back to this 😅 )
I guess my main problem with this is that we do all the work of parsing the "old" request etc. even though we don't need it? I do feel like it would be easier to understand to have a single function be responsible for it - otherwise, we end up in the state of "why is this field set here? let's look through all of the old code to find if it sets it somewhere, then look over the new code, ...".
By not calling the old method anymore it makes it easier to follow the "new, happy" path imho 🤔 though I do agree that the additionalData part is not great...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@lforst ? 🙃
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@maximedupre we will get to it. Please understand that we are bound to other obligations, absences, deadlines... Pinging usually doesn't influence in what order we do things so I would generally just not do it - in any open source project. Thanks!
🥹 |
183be5b
to
a5821ce
Compare
a5821ce
to
3ad5459
Compare
edab56c
to
ae94190
Compare
packages/node/src/integrations/http/SentryHttpInstrumentation.ts
Outdated
Show resolved
Hide resolved
packages/node/src/integrations/http/SentryHttpInstrumentation.ts
Outdated
Show resolved
Hide resolved
packages/node/src/integrations/http/SentryHttpInstrumentation.ts
Outdated
Show resolved
Hide resolved
/** | ||
* Request data included in an event as sent to Sentry. | ||
* TODO(v9): Rename this to avoid confusion, because Request is also a native type. | ||
*/ | ||
export interface Request { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
l: Can't we change this now? What exposed to users relies on this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it is exported from @sentry/types
, probably does not affect anybody, but 🤷 IMHO I would just clean this up and rename these things in v9, WDYT?
383e4f2
to
28e6dfe
Compare
Co-authored-by: Luca Forstner <[email protected]>
28e6dfe
to
db8c688
Compare
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [@sentry/node](https://github.com/getsentry/sentry-javascript/tree/master/packages/node) ([source](https://github.com/getsentry/sentry-javascript)) | dependencies | minor | [`8.37.1` -> `8.40.0`](https://renovatebot.com/diffs/npm/@sentry%2fnode/8.37.1/8.40.0) | | [@sentry/react](https://github.com/getsentry/sentry-javascript/tree/master/packages/react) ([source](https://github.com/getsentry/sentry-javascript)) | dependencies | minor | [`8.37.1` -> `8.40.0`](https://renovatebot.com/diffs/npm/@sentry%2freact/8.37.1/8.40.0) | --- ### Release Notes <details> <summary>getsentry/sentry-javascript (@​sentry/node)</summary> ### [`v8.40.0`](https://github.com/getsentry/sentry-javascript/blob/HEAD/CHANGELOG.md#8400) [Compare Source](getsentry/sentry-javascript@8.39.0...8.40.0) ##### Important Changes - **feat(angular): Support Angular 19 ([#​14398](getsentry/sentry-javascript#14398 The `@sentry/angular` SDK can now be used with Angular 19. If you're upgrading to the new Angular version, you might want to migrate from the now deprecated `APP_INITIALIZER` token to `provideAppInitializer`. In this case, change the Sentry `TraceService` initialization in `app.config.ts`: ```ts // Angular 18 export const appConfig: ApplicationConfig = { providers: [ // other providers { provide: TraceService, deps: [Router], }, { provide: APP_INITIALIZER, useFactory: () => () => {}, deps: [TraceService], multi: true, }, ], }; // Angular 19 export const appConfig: ApplicationConfig = { providers: [ // other providers { provide: TraceService, deps: [Router], }, provideAppInitializer(() => { inject(TraceService); }), ], }; ``` - **feat(core): Deprecate `debugIntegration` and `sessionTimingIntegration` ([#​14363](getsentry/sentry-javascript#14363 The `debugIntegration` was deprecated and will be removed in the next major version of the SDK. To log outgoing events, use [Hook Options](https://docs.sentry.io/platforms/javascript/configuration/options/#hooks) (`beforeSend`, `beforeSendTransaction`, ...). The `sessionTimingIntegration` was deprecated and will be removed in the next major version of the SDK. To capture session durations alongside events, use [Context](https://docs.sentry.io/platforms/javascript/enriching-events/context/) (`Sentry.setContext()`). - **feat(nestjs): Deprecate `@WithSentry` in favor of `@SentryExceptionCaptured` ([#​14323](getsentry/sentry-javascript#14323 The `@WithSentry` decorator was deprecated. Use `@SentryExceptionCaptured` instead. This is a simple renaming and functionality stays identical. - **feat(nestjs): Deprecate `SentryTracingInterceptor`, `SentryService`, `SentryGlobalGenericFilter`, `SentryGlobalGraphQLFilter` ([#​14371](getsentry/sentry-javascript#14371 The `SentryTracingInterceptor` was deprecated. If you are using `@sentry/nestjs` you can safely remove any references to the `SentryTracingInterceptor`. If you are using another package migrate to `@sentry/nestjs` and remove the `SentryTracingInterceptor` afterwards. The `SentryService` was deprecated and its functionality was added to `Sentry.init`. If you are using `@sentry/nestjs` you can safely remove any references to the `SentryService`. If you are using another package migrate to `@sentry/nestjs` and remove the `SentryService` afterwards. The `SentryGlobalGenericFilter` was deprecated. Use the `SentryGlobalFilter` instead which is a drop-in replacement. The `SentryGlobalGraphQLFilter` was deprecated. Use the `SentryGlobalFilter` instead which is a drop-in replacement. - **feat(node): Deprecate `nestIntegration` and `setupNestErrorHandler` in favor of using `@sentry/nestjs` ([#​14374](getsentry/sentry-javascript#14374 The `nestIntegration` and `setupNestErrorHandler` functions from `@sentry/node` were deprecated and will be removed in the next major version of the SDK. If you're using `@sentry/node` in a NestJS application, we recommend switching to our new dedicated `@sentry/nestjs` package. ##### Other Changes - feat(browser): Send additional LCP timing info ([#​14372](getsentry/sentry-javascript#14372)) - feat(replay): Clear event buffer when full and in buffer mode ([#​14078](getsentry/sentry-javascript#14078)) - feat(core): Ensure `normalizedRequest` on `sdkProcessingMetadata` is merged ([#​14315](getsentry/sentry-javascript#14315)) - feat(core): Hoist everything from `@sentry/utils` into `@sentry/core` ([#​14382](getsentry/sentry-javascript#14382)) - fix(core): Do not throw when trying to fill readonly properties ([#​14402](getsentry/sentry-javascript#14402)) - fix(feedback): Fix `__self` and `__source` attributes on feedback nodes ([#​14356](getsentry/sentry-javascript#14356)) - fix(feedback): Fix non-wrapping form title ([#​14355](getsentry/sentry-javascript#14355)) - fix(nextjs): Update check for not found navigation error ([#​14378](getsentry/sentry-javascript#14378)) ### [`v8.39.0`](https://github.com/getsentry/sentry-javascript/blob/HEAD/CHANGELOG.md#8390) [Compare Source](getsentry/sentry-javascript@8.38.0...8.39.0) ##### Important Changes - **feat(nestjs): Instrument event handlers ([#​14307](getsentry/sentry-javascript#14307 The `@sentry/nestjs` SDK will now capture performance data for [NestJS Events (`@nestjs/event-emitter`)](https://docs.nestjs.com/techniques/events) ##### Other Changes - feat(nestjs): Add alias `@SentryExceptionCaptured` for `@WithSentry` ([#​14322](getsentry/sentry-javascript#14322)) - feat(nestjs): Duplicate `SentryService` behaviour into `@sentry/nestjs` SDK `init()` ([#​14321](getsentry/sentry-javascript#14321)) - feat(nestjs): Handle GraphQL contexts in `SentryGlobalFilter` ([#​14320](getsentry/sentry-javascript#14320)) - feat(node): Add alias `childProcessIntegration` for `processThreadBreadcrumbIntegration` and deprecate it ([#​14334](getsentry/sentry-javascript#14334)) - feat(node): Ensure request bodies are reliably captured for http requests ([#​13746](getsentry/sentry-javascript#13746)) - feat(replay): Upgrade rrweb packages to 2.29.0 ([#​14160](getsentry/sentry-javascript#14160)) - fix(cdn): Ensure `_sentryModuleMetadata` is not mangled ([#​14344](getsentry/sentry-javascript#14344)) - fix(core): Set `sentry.source` attribute to `custom` when calling `span.updateName` on `SentrySpan` ([#​14251](getsentry/sentry-javascript#14251)) - fix(mongo): rewrite Buffer as ? during serialization ([#​14071](getsentry/sentry-javascript#14071)) - fix(replay): Remove replay id from DSC on expired sessions ([#​14342](getsentry/sentry-javascript#14342)) - ref(profiling) Fix electron crash ([#​14216](getsentry/sentry-javascript#14216)) - ref(types): Deprecate `Request` type in favor of `RequestEventData` ([#​14317](getsentry/sentry-javascript#14317)) - ref(utils): Stop setting `transaction` in `requestDataIntegration` ([#​14306](getsentry/sentry-javascript#14306)) - ref(vue): Reduce bundle size for starting application render span ([#​14275](getsentry/sentry-javascript#14275)) ### [`v8.38.0`](https://github.com/getsentry/sentry-javascript/blob/HEAD/CHANGELOG.md#8380) [Compare Source](getsentry/sentry-javascript@8.37.1...8.38.0) - docs: Improve docstrings for node otel integrations ([#​14217](getsentry/sentry-javascript#14217)) - feat(browser): Add moduleMetadataIntegration lazy loading support ([#​13817](getsentry/sentry-javascript#13817)) - feat(core): Add trpc path to context in trpcMiddleware ([#​14218](getsentry/sentry-javascript#14218)) - feat(deps): Bump [@​opentelemetry/instrumentation-amqplib](https://github.com/opentelemetry/instrumentation-amqplib) from 0.42.0 to 0.43.0 ([#​14230](getsentry/sentry-javascript#14230)) - feat(deps): Bump [@​sentry/cli](https://github.com/sentry/cli) from 2.37.0 to 2.38.2 ([#​14232](getsentry/sentry-javascript#14232)) - feat(node): Add `knex` integration ([#​13526](getsentry/sentry-javascript#13526)) - feat(node): Add `tedious` integration ([#​13486](getsentry/sentry-javascript#13486)) - feat(utils): Single implementation to fetch debug ids ([#​14199](getsentry/sentry-javascript#14199)) - fix(browser): Avoid recording long animation frame spans starting before their parent span ([#​14186](getsentry/sentry-javascript#14186)) - fix(node): Include `debug_meta` with ANR events ([#​14203](getsentry/sentry-javascript#14203)) - fix(nuxt): Fix dynamic import rollup plugin to work with latest nitro ([#​14243](getsentry/sentry-javascript#14243)) - fix(react): Support wildcard routes on React Router 6 ([#​14205](getsentry/sentry-javascript#14205)) - fix(spotlight): Export spotlightBrowserIntegration from the main browser package ([#​14208](getsentry/sentry-javascript#14208)) - ref(browser): Ensure start time of interaction root and child span is aligned ([#​14188](getsentry/sentry-javascript#14188)) - ref(nextjs): Make build-time value injection turbopack compatible ([#​14081](getsentry/sentry-javascript#14081)) Work in this release was contributed by [@​grahamhency](https://github.com/grahamhency), [@​Zen-cronic](https://github.com/Zen-cronic), [@​gilisho](https://github.com/gilisho) and [@​phuctm97](https://github.com/phuctm97). Thank you for your contributions! </details> --- ### Configuration 📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC4xNDIuNyIsInVwZGF0ZWRJblZlciI6IjM4LjE0Mi43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJkZXBlbmRlbmNpZXMiXX0=--> Reviewed-on: https://git.tristess.app/alexandresoro/ouca/pulls/317 Reviewed-by: Alexandre Soro <[email protected]> Co-authored-by: renovate <[email protected]> Co-committed-by: renovate <[email protected]>
This PR started out as trying to fix capturing request bodies for Koa.
Some investigation later, we found out that the fundamental problem was that we relied on the request body being on
request.body
, which is non-standard and thus does not necessarily works. It seems that in express this works because it under the hood writes the body there, but this is non-standard and rather undefined behavior. For other frameworks (e.g. Koa and probably more) this did not work, the body was not on the request and thus never captured. We also had no test coverage for this overall.This PR ended up doing a few things:
sdkProcessingMetadata
- this used to beany
, which lead to any usage of this not really being typed at all. I added proper types for this now.Most importantly, I opted to not force this into the existing, rather complicated and hard to follow request data integration flow. This used to take an IsomorphicRequest and then did a bunch of conversion etc.
Since now in Node, we always have the same, proper http request (for any framework, because this always goes through http instrumentation), we can actually streamline this and normalize this properly at the time where we set this.
So with this PR, we set a
normalizedRequest
which already has the url, headers etc. set in a way that we need it/it makes sense.Additionally, the parsed & stringified request body will be set on this too.
If this normalized request is set in sdkProcessingMetadata, we will use it as source of truth instead of the plain
request
. (Note that we still need the plain request for some auxiliary data that is non-standard, e.g.request.user
).For the body parsing itself, we monkey-patch
req.on('data')
. this way, we ensure to not add more handlers than a user has, and we only extract the body if the user is extracting it anyhow, ensuring we do not alter behavior.Closes #13722