feat(devserver/http): support gzipped responses #4011

kwonoj · 2023-02-28T18:38:26Z

Implements WEB-666.

This PR enables gzip compressions for the responses by default. This can be extended to other compression method (i.e brotli), but for now implements minimal default compression only to pass existing test case.

Fixes test/integration/compression/test/index.test.js.

vercel · 2023-02-28T18:38:38Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated
examples-designsystem-docs	🔄 Building (Inspect)			Mar 2, 2023 at 5:15AM (UTC)

9 Ignored Deployments

Name	Status	Preview	Updated
examples-basic-web	⬜️ Ignored (Inspect)		Mar 2, 2023 at 5:15AM (UTC)
examples-cra-web	⬜️ Ignored (Inspect)		Mar 2, 2023 at 5:15AM (UTC)
examples-kitchensink-blog	⬜️ Ignored (Inspect)		Mar 2, 2023 at 5:15AM (UTC)
examples-native-web	⬜️ Ignored (Inspect)		Mar 2, 2023 at 5:15AM (UTC)
examples-nonmonorepo	⬜️ Ignored (Inspect)		Mar 2, 2023 at 5:15AM (UTC)
examples-svelte-web	⬜️ Ignored (Inspect)		Mar 2, 2023 at 5:15AM (UTC)
examples-tailwind-web	⬜️ Ignored (Inspect)		Mar 2, 2023 at 5:15AM (UTC)
examples-vite-web	⬜️ Ignored (Inspect)		Mar 2, 2023 at 5:15AM (UTC)
turbo-site	⬜️ Ignored (Inspect)	Visit Preview	Mar 2, 2023 at 5:15AM (UTC)

github-actions · 2023-02-28T19:08:11Z

🟢 CI successful 🟢

Thanks

github-actions · 2023-02-28T20:30:49Z

Benchmark for `072559e`

Click to view benchmark

Test	Base	PR	%
bench_hmr_to_commit/Turbopack CSR/1000 modules	8312.53µs ± 91.23µs	8230.31µs ± 58.48µs	-0.99%
bench_hmr_to_commit/Turbopack RCC/1000 modules	13.22ms ± 0.11ms	13.31ms ± 0.08ms	+0.69%
bench_hmr_to_commit/Turbopack RSC/1000 modules	478.53ms ± 1.15ms	481.09ms ± 0.95ms	+0.54%
bench_hmr_to_commit/Turbopack SSR/1000 modules	8706.60µs ± 140.65µs	8600.97µs ± 62.90µs	-1.21%
bench_hmr_to_eval/Turbopack CSR/1000 modules	7619.11µs ± 103.63µs	7468.44µs ± 38.10µs	-1.98%
bench_hmr_to_eval/Turbopack RCC/1000 modules	12.04ms ± 0.16ms	12.38ms ± 0.19ms	+2.83%
bench_hmr_to_eval/Turbopack SSR/1000 modules	7647.35µs ± 77.27µs	7571.36µs ± 38.38µs	-0.99%
bench_hydration/Turbopack RCC/1000 modules	3745.97ms ± 8.17ms	3754.25ms ± 11.36ms	+0.22%
bench_hydration/Turbopack RSC/1000 modules	3391.51ms ± 10.35ms	3400.82ms ± 15.93ms	+0.27%
bench_hydration/Turbopack SSR/1000 modules	3302.71ms ± 10.80ms	3329.26ms ± 11.36ms	+0.80%
bench_startup/Turbopack CSR/1000 modules	2512.84ms ± 5.32ms	2520.10ms ± 8.28ms	+0.29%
bench_startup/Turbopack RCC/1000 modules	2332.75ms ± 13.78ms	2329.78ms ± 7.72ms	-0.13%
bench_startup/Turbopack RSC/1000 modules	2238.10ms ± 9.36ms	2229.48ms ± 13.02ms	-0.39%
bench_startup/Turbopack SSR/1000 modules	2025.27ms ± 5.15ms	2032.74ms ± 2.24ms	+0.37%

jridgewell · 2023-03-01T01:40:32Z

crates/turbopack-dev-server/src/http.rs

+                let stream_ext = content
+                    .read()
+                    .into_stream()
+                    .map_err(|err| std::io::Error::new(std::io::ErrorKind::Other, err));


Nit: (with appropriate use std::io; statement at the top)

Suggested change

.map_err(|err| std::io::Error::new(std::io::ErrorKind::Other, err));

.map_err(|err| io::Error::new(io::ErrorKind::Other, err));

jridgewell · 2023-03-01T01:41:23Z

crates/turbopack-dev-server/src/http.rs

+                let gzipped_stream = tokio_util::io::ReaderStream::new(
+                    async_compression::tokio::bufread::GzipEncoder::new(
+                        tokio_util::io::StreamReader::new(stream_ext),


Nit: with use tokio_util::io::{ReaderStream, StreamReader}; use async_compression::tokio::bufread::GzipEncoder;

Suggested change

let gzipped_stream = tokio_util::io::ReaderStream::new(

async_compression::tokio::bufread::GzipEncoder::new(

tokio_util::io::StreamReader::new(stream_ext),

let gzipped_stream = ReaderStream::new(

GzipEncoder::new(

StreamReader::new(stream_ext),

jridgewell · 2023-03-01T01:59:46Z

crates/turbopack-dev-server/src/http.rs

+                    .into_stream()
+                    .map_err(|err| std::io::Error::new(std::io::ErrorKind::Other, err));
+
+                let gzipped_stream = tokio_util::io::ReaderStream::new(


Not all content should be compressed, mainly images except SVG. Compressing binary formats will only increase the file size. Can we do an inspection for the content-type to ensure it's a textual format?

Probably yes, could have some basic mime check for the content types.

Express's compression middleware uses compressible

yes, that's what I figured. But we should have rust corresponding one instead, isn't it?

The rust crate seems out of date? We could get most of the way by just using the fallback algorithm from compressible:

text/*

*/*+json

*/*+text

*/*+xml

and a few extras for application/json and application/javascript

You mean only compress for those, correct?

github-actions · 2023-03-01T04:41:09Z

Benchmark for `6434755`

Click to view benchmark

Test	Base	PR	%
bench_hmr_to_commit/Turbopack CSR/1000 modules	9211.25µs ± 51.97µs	9137.45µs ± 46.04µs	-0.80%
bench_hmr_to_commit/Turbopack RCC/1000 modules	12.93ms ± 0.10ms	12.83ms ± 0.23ms	-0.74%
bench_hmr_to_commit/Turbopack RSC/1000 modules	501.09ms ± 1.21ms	503.01ms ± 1.59ms	+0.38%
bench_hmr_to_commit/Turbopack SSR/1000 modules	9413.75µs ± 34.81µs	9337.89µs ± 43.47µs	-0.81%
bench_hmr_to_eval/Turbopack CSR/1000 modules	8245.61µs ± 44.03µs	8222.30µs ± 47.22µs	-0.28%
bench_hmr_to_eval/Turbopack RCC/1000 modules	11.42ms ± 0.20ms	11.16ms ± 0.27ms	-2.35%
bench_hmr_to_eval/Turbopack SSR/1000 modules	8295.97µs ± 54.44µs	8270.82µs ± 45.10µs	-0.30%
bench_hydration/Turbopack RCC/1000 modules	3804.32ms ± 12.11ms	3824.05ms ± 13.99ms	+0.52%
bench_hydration/Turbopack RSC/1000 modules	3470.14ms ± 8.99ms	3459.48ms ± 12.99ms	-0.31%
bench_hydration/Turbopack SSR/1000 modules	3337.12ms ± 13.00ms	3370.97ms ± 19.19ms	+1.01%
bench_startup/Turbopack CSR/1000 modules	2551.27ms ± 8.28ms	2556.87ms ± 11.24ms	+0.22%
bench_startup/Turbopack RCC/1000 modules	2352.05ms ± 6.96ms	2361.52ms ± 8.52ms	+0.40%
bench_startup/Turbopack RSC/1000 modules	2282.98ms ± 4.55ms	2285.47ms ± 10.34ms	+0.11%
bench_startup/Turbopack SSR/1000 modules	2044.48ms ± 4.34ms	2045.52ms ± 5.34ms	+0.05%

jridgewell · 2023-03-02T05:06:58Z

crates/turbopack-dev-server/src/http.rs

@@ -92,14 +100,28 @@ pub async fn process_request_with_content_source(
                    );
                }

+                // naively checking if content is `compressible`.
+                let mut should_compress = false;
+                let should_compress_predicate = |mime: &Mime| match (mime.type_(), mime.subtype()) {


We should also be checking mime.suffix() for:

xml

json

text

github-actions · 2023-03-02T06:32:26Z

Benchmark for `3d84ec8`

Test	Base	PR	%	Significant %
bench_hmr_to_commit/Turbopack RCC/1000 modules	12.65ms ± 0.22ms	13.58ms ± 0.11ms	+7.36%	+2.03%
bench_hmr_to_eval/Turbopack CSR/1000 modules	9107.58µs ± 32.53µs	8854.76µs ± 39.11µs	-2.78%	-1.21%

Click to view full benchmark

Test	Base	PR	%	Significant %
bench_hmr_to_commit/Turbopack CSR/1000 modules	10.02ms ± 0.03ms	10.04ms ± 0.03ms	+0.24%
bench_hmr_to_commit/Turbopack RCC/1000 modules	12.65ms ± 0.22ms	13.58ms ± 0.11ms	+7.36%	+2.03%
bench_hmr_to_commit/Turbopack RSC/1000 modules	527.31ms ± 5.97ms	530.08ms ± 1.41ms	+0.52%
bench_hmr_to_commit/Turbopack SSR/1000 modules	9922.65µs ± 80.47µs	10.03ms ± 0.05ms	+1.09%
bench_hmr_to_eval/Turbopack CSR/1000 modules	9107.58µs ± 32.53µs	8854.76µs ± 39.11µs	-2.78%	-1.21%
bench_hmr_to_eval/Turbopack RCC/1000 modules	11.36ms ± 0.08ms	11.34ms ± 0.06ms	-0.24%
bench_hmr_to_eval/Turbopack SSR/1000 modules	9212.45µs ± 49.81µs	9219.34µs ± 67.94µs	+0.07%
bench_hydration/Turbopack RCC/1000 modules	3920.32ms ± 10.18ms	3954.29ms ± 13.53ms	+0.87%
bench_hydration/Turbopack RSC/1000 modules	3578.47ms ± 11.89ms	3546.35ms ± 14.38ms	-0.90%
bench_hydration/Turbopack SSR/1000 modules	3413.51ms ± 8.80ms	3410.29ms ± 13.54ms	-0.09%
bench_startup/Turbopack CSR/1000 modules	2610.01ms ± 8.17ms	2628.34ms ± 5.90ms	+0.70%
bench_startup/Turbopack RCC/1000 modules	2359.71ms ± 7.28ms	2349.47ms ± 5.93ms	-0.43%
bench_startup/Turbopack RSC/1000 modules	2295.50ms ± 7.00ms	2302.86ms ± 8.12ms	+0.32%
bench_startup/Turbopack SSR/1000 modules	2086.03ms ± 6.19ms	2085.01ms ± 4.14ms	-0.05%

# New Features * vercel/turborepo#4011 # Performance Improvements * vercel/turborepo#3955 * vercel/turborepo#4018 # Bug Fixes * vercel/turborepo#4037 * vercel/turborepo#4028 # Other * vercel/turborepo#3974 * vercel/turborepo#4015 * vercel/turborepo#3999 * vercel/turborepo#4026 * vercel/turborepo#4053 * vercel/turborepo#3891 Co-authored-by: JJ Kasper <[email protected]>

Implements WEB-666. This PR enables gzip compressions for the responses by default. This can be extended to other compression method (i.e brotli), but for now implements minimal default compression only to pass existing test case. Fixes `test/integration/compression/test/index.test.js`.

kwonoj requested review from a team as code owners February 28, 2023 18:38

kwonoj requested review from mehulkar and arlyon February 28, 2023 18:38

turbo-orchestrator bot added team: turbopack labels Feb 28, 2023

kwonoj force-pushed the feat-gzipped-response-turbo-dev branch from b34ae77 to 029759c Compare February 28, 2023 19:14

mehulkar removed their request for review February 28, 2023 21:30

jridgewell reviewed Mar 1, 2023

View reviewed changes

kwonoj added 3 commits February 28, 2023 19:06

build(cargo): update dependencies

f9197aa

feat(devserver/http): support gzipped responses

e8f98b9

build(cargo): update lockfile

9cbcd8d

kwonoj force-pushed the feat-gzipped-response-turbo-dev branch from 029759c to 0a6892f Compare March 1, 2023 03:27

vercel bot deployed to Preview – turbo-site March 1, 2023 03:28 View deployment

jridgewell reviewed Mar 2, 2023

View reviewed changes

feat(devserver/http): naively check content type for compression

a88aefb

kwonoj force-pushed the feat-gzipped-response-turbo-dev branch from 0a6892f to a88aefb Compare March 2, 2023 05:14

jridgewell approved these changes Mar 2, 2023

View reviewed changes

kwonoj added the pr: automerge Kodiak will merge these automatically after checks pass label Mar 2, 2023

kodiakhq bot merged commit b6dc9d0 into main Mar 2, 2023

kodiakhq bot deleted the feat-gzipped-response-turbo-dev branch March 2, 2023 05:39

alexkirsz mentioned this pull request Mar 2, 2023

build(cargo): update turbopack vercel/next.js#46685

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(devserver/http): support gzipped responses #4011

feat(devserver/http): support gzipped responses #4011

kwonoj commented Feb 28, 2023

vercel bot commented Feb 28, 2023 •

edited

Loading

github-actions bot commented Feb 28, 2023 •

edited

Loading

github-actions bot commented Feb 28, 2023

jridgewell Mar 1, 2023

jridgewell Mar 1, 2023

jridgewell Mar 1, 2023

kwonoj Mar 1, 2023

jridgewell Mar 1, 2023

kwonoj Mar 1, 2023 •

edited

Loading

jridgewell Mar 1, 2023 •

edited

Loading

kwonoj Mar 1, 2023

jridgewell Mar 1, 2023

github-actions bot commented Mar 1, 2023

jridgewell Mar 2, 2023

github-actions bot commented Mar 2, 2023

	.map_err(\|err\| std::io::Error::new(std::io::ErrorKind::Other, err));
	.map_err(\|err\| io::Error::new(io::ErrorKind::Other, err));

feat(devserver/http): support gzipped responses #4011

feat(devserver/http): support gzipped responses #4011

Conversation

kwonoj commented Feb 28, 2023

vercel bot commented Feb 28, 2023 • edited Loading

github-actions bot commented Feb 28, 2023 • edited Loading

🟢 CI successful 🟢

github-actions bot commented Feb 28, 2023

Benchmark for 072559e

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kwonoj Mar 1, 2023 • edited Loading

Choose a reason for hiding this comment

jridgewell Mar 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Mar 1, 2023

Benchmark for 6434755

Choose a reason for hiding this comment

github-actions bot commented Mar 2, 2023

Benchmark for 3d84ec8

vercel bot commented Feb 28, 2023 •

edited

Loading

github-actions bot commented Feb 28, 2023 •

edited

Loading

Benchmark for `072559e`

kwonoj Mar 1, 2023 •

edited

Loading

jridgewell Mar 1, 2023 •

edited

Loading

Benchmark for `6434755`

Benchmark for `3d84ec8`