[Turbopack] use thread local trace collecting

[sokra]

What?

Collect trace events in a thread local buffer first and then send a batch of events to the writer thread.

But there are some challenges to that:

Events might be out of order in the trace file. Usually that should be fine since they have a timestamp, but

• span ids might be reused faster than all events are send to the file. So we assign our own globally unique ids instead.
• the trace server need to be able to process events out of order. So we queue up events when they are received for spans that are not created yet.

And the thread local buffer might also have a problem when the thread is idle or hanging. This would cause events to never be written to the file.
To fix that the writer thread will steal the outstanding events from the thread local buffers when it is idle for a second. This ensures that all events are eventually written, even in case of hanging threads.

Closes PACK-3631

[sokra]

• [Turbopack] use thread local trace collecting #73615 👈 (View in Graphite)
• [Turbopack] replace EnvLayer with a faster filter #73614
• canary

This stack of pull requests is managed by Graphite. Learn more about stacking.

[ijjk]

Stats from current PR

Default Build (Increase detected ⚠️)

General

	vercel/next.js canary	vercel/next.js sokra/perf-tracing-2	Change
buildDuration	20.9s	17.5s	N/A
buildDurationCached	16.7s	14.7s	N/A
nodeModulesSize	409 MB	409 MB	N/A
nextStartRea..uration (ms)	472ms	468ms	N/A

Client Bundles (main, webpack)

	vercel/next.js canary	vercel/next.js sokra/perf-tracing-2	Change
1187-HASH.js gzip	50.2 kB	50.2 kB	N/A
8276.HASH.js gzip	169 B	168 B	N/A
8377-HASH.js gzip	5.3 kB	5.3 kB	N/A
bccd1874-HASH.js gzip	53 kB	53 kB	N/A
framework-HASH.js gzip	57.5 kB	57.5 kB	N/A
main-app-HASH.js gzip	233 B	235 B	N/A
main-HASH.js gzip	33.8 kB	33.7 kB	N/A
webpack-HASH.js gzip	1.71 kB	1.71 kB	N/A
Overall change	0 B	0 B	✓

Legacy Client Bundles (polyfills)

	vercel/next.js canary	vercel/next.js sokra/perf-tracing-2	Change
polyfills-HASH.js gzip	39.4 kB	39.4 kB	✓
Overall change	39.4 kB	39.4 kB	✓

Client Pages

	vercel/next.js canary	vercel/next.js sokra/perf-tracing-2	Change
_app-HASH.js gzip	193 B	193 B	✓
_error-HASH.js gzip	193 B	193 B	✓
amp-HASH.js gzip	512 B	510 B	N/A
css-HASH.js gzip	343 B	342 B	N/A
dynamic-HASH.js gzip	1.84 kB	1.84 kB	✓
edge-ssr-HASH.js gzip	265 B	265 B	✓
head-HASH.js gzip	363 B	362 B	N/A
hooks-HASH.js gzip	393 B	392 B	N/A
image-HASH.js gzip	4.44 kB	4.43 kB	N/A
index-HASH.js gzip	268 B	268 B	✓
link-HASH.js gzip	2.35 kB	2.34 kB	N/A
routerDirect..HASH.js gzip	328 B	328 B	✓
script-HASH.js gzip	397 B	397 B	✓
withRouter-HASH.js gzip	323 B	326 B	N/A
1afbb74e6ecf..834.css gzip	106 B	106 B	✓
Overall change	3.59 kB	3.59 kB	✓

Client Build Manifests

	vercel/next.js canary	vercel/next.js sokra/perf-tracing-2	Change
_buildManifest.js gzip	747 B	745 B	N/A
Overall change	0 B	0 B	✓

Rendered Page Sizes

	vercel/next.js canary	vercel/next.js sokra/perf-tracing-2	Change
index.html gzip	524 B	524 B	✓
link.html gzip	539 B	539 B	✓
withRouter.html gzip	520 B	521 B	N/A
Overall change	1.06 kB	1.06 kB	✓

Edge SSR bundle Size

	vercel/next.js canary	vercel/next.js sokra/perf-tracing-2	Change
edge-ssr.js gzip	128 kB	128 kB	N/A
page.js gzip	203 kB	203 kB	N/A
Overall change	0 B	0 B	✓

Middleware size

	vercel/next.js canary	vercel/next.js sokra/perf-tracing-2	Change
middleware-b..fest.js gzip	669 B	667 B	N/A
middleware-r..fest.js gzip	155 B	156 B	N/A
middleware.js gzip	31 kB	31 kB	N/A
edge-runtime..pack.js gzip	844 B	844 B	✓
Overall change	844 B	844 B	✓

Next Runtimes

	vercel/next.js canary	vercel/next.js sokra/perf-tracing-2	Change
523-experime...dev.js gzip	322 B	322 B	✓
523.runtime.dev.js gzip	314 B	314 B	✓
app-page-exp...dev.js gzip	322 kB	322 kB	✓
app-page-exp..prod.js gzip	127 kB	127 kB	✓
app-page-tur..prod.js gzip	140 kB	140 kB	✓
app-page-tur..prod.js gzip	135 kB	135 kB	✓
app-page.run...dev.js gzip	312 kB	312 kB	✓
app-page.run..prod.js gzip	122 kB	122 kB	✓
app-route-ex...dev.js gzip	37.1 kB	37.1 kB	✓
app-route-ex..prod.js gzip	25.1 kB	25.1 kB	✓
app-route-tu..prod.js gzip	25.1 kB	25.1 kB	✓
app-route-tu..prod.js gzip	24.9 kB	24.9 kB	✓
app-route.ru...dev.js gzip	38.7 kB	38.7 kB	✓
app-route.ru..prod.js gzip	24.9 kB	24.9 kB	✓
pages-api-tu..prod.js gzip	9.56 kB	9.56 kB	✓
pages-api.ru...dev.js gzip	11.4 kB	11.4 kB	✓
pages-api.ru..prod.js gzip	9.56 kB	9.56 kB	✓
pages-turbo...prod.js gzip	21.3 kB	21.3 kB	✓
pages.runtim...dev.js gzip	27 kB	27 kB	✓
pages.runtim..prod.js gzip	21.3 kB	21.3 kB	✓
server.runti..prod.js gzip	916 kB	916 kB	✓
Overall change	2.35 MB	2.35 MB	✓

build cache Overall increase ⚠️

	vercel/next.js canary	vercel/next.js sokra/perf-tracing-2	Change
0.pack gzip	2.03 MB	2.04 MB	⚠️ +2.85 kB
index.pack gzip	145 kB	145 kB	⚠️ +641 B
Overall change	2.18 MB	2.18 MB	⚠️ +3.49 kB

Diff details

Diff for main-HASH.js

Diff too large to display

Commit: 8d0272e

[ijjk]

Tests Passed

[sokra]

Merge activity

• Dec 10, 2:53 AM EST: A user merged this pull request with Graphite.