Refactor prerendering chunk handling #11245

bluwy · 2024-06-12T14:31:56Z

Changes

NOTE: This is a big PR but mostly due to test changes/additions.

This PR changes how chunks are handled for prerendering. Before:

For each page, the page.route.prerender metadata is derived in manualChunks (slow)
If page.route.prerender is true, it's grouped into a special "prerender" chunk.
After prerendering, we search for the "prerender" chunk and mark every export with a no-op.
PROBLEM: The "prerender" chunk may contain non-prerender code due to Rollup chunking, where it thinks certain code are used together and group them in a single chunk regardless.

After:

For each page, the page.route.prerender metadata is derived in generateBundle (faster)
During generateBundle, we figure out chunks that are used in prerender-only, and track them to be deleted at the end.
To make the chunk deletion efficient, each page has their corresponding Rollup input where we can be sure of deleting the input directly after building.

Testing

It includes test from #10738

There are also a lot of changes related to outDir, because each page are now its own input chunk, when we build fixtures multiple times, ESM could cache the input chunk path, causing subsequent fixtures runtime to use the previous, which is problematic also for CCC.

Docs

n/a. Mostly a bug fix and it shouldn't affect Astro projects in practice.

changeset-bot · 2024-06-12T14:32:01Z

🦋 Changeset detected

Latest commit: d0bd620

The changes in this PR will be included in the next version bump.

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

bluwy

Left some review notes below

packages/astro/src/core/build/plugins/plugin-chunks.ts

packages/astro/src/core/build/plugins/plugin-pages.ts

packages/astro/src/core/build/plugins/plugin-prerender.ts

packages/astro/src/core/build/plugins/plugin-ssr.ts

packages/astro/test/astro-assets-prefix.test.js

packages/astro/test/ssr-prerender.test.js

packages/astro/src/core/build/plugins/plugin-prerender.ts

packages/astro/test/astro-assets-prefix.test.js

ematipico

Awesome work @bluwy !

bluwy · 2024-06-14T14:06:29Z

!preview refactor-prerender

github-actions · 2024-06-14T14:09:18Z

Snapshots have been released for the following packages:

astro@experimental--refactor-prerender
@astrojs/vercel@experimental--refactor-prerender

Publish Log

🦋  warn ===============================IMPORTANT!===============================
🦋  warn Packages will be released under the experimental--refactor-prerender tag
🦋  warn ----------------------------------------------------------------------
🦋  info npm info astro
🦋  info npm info @astrojs/prism
🦋  info npm info @astrojs/rss
🦋  info npm info create-astro
🦋  info npm info @astrojs/db
🦋  info npm info @astrojs/alpinejs
🦋  info npm info @astrojs/lit
🦋  info npm info @astrojs/markdoc
🦋  info npm info @astrojs/mdx
🦋  info npm info @astrojs/node
🦋  info npm info @astrojs/partytown
🦋  info npm info @astrojs/preact
🦋  info npm info @astrojs/react
🦋  info npm info @astrojs/sitemap
🦋  info npm info @astrojs/solid-js
🦋  info npm info @astrojs/svelte
🦋  info npm info @astrojs/tailwind
🦋  info npm info @astrojs/vercel
🦋  info npm info @astrojs/vue
🦋  info npm info @astrojs/web-vitals
🦋  info npm info @astrojs/internal-helpers
🦋  info npm info @astrojs/markdown-remark
🦋  info npm info @astrojs/studio
🦋  info npm info @astrojs/telemetry
🦋  info npm info @astrojs/underscore-redirects
🦋  info npm info @astrojs/upgrade
🦋  info astro is being published because our local version (0.0.0-refactor-prerender-20240614140807) has not been published on npm
🦋  warn @astrojs/prism is not being published because version 3.1.0 is already published on npm
🦋  warn @astrojs/rss is not being published because version 4.0.6 is already published on npm
🦋  warn create-astro is not being published because version 4.8.0 is already published on npm
🦋  warn @astrojs/db is not being published because version 0.11.5 is already published on npm
🦋  warn @astrojs/alpinejs is not being published because version 0.4.0 is already published on npm
🦋  warn @astrojs/lit is not being published because version 4.2.0 is already published on npm
🦋  warn @astrojs/markdoc is not being published because version 0.11.0 is already published on npm
🦋  warn @astrojs/mdx is not being published because version 3.1.0 is already published on npm
🦋  warn @astrojs/node is not being published because version 8.3.0 is already published on npm
🦋  warn @astrojs/partytown is not being published because version 2.1.1 is already published on npm
🦋  warn @astrojs/preact is not being published because version 3.4.0 is already published on npm
🦋  warn @astrojs/react is not being published because version 3.5.0 is already published on npm
🦋  warn @astrojs/sitemap is not being published because version 3.1.5 is already published on npm
🦋  warn @astrojs/solid-js is not being published because version 4.3.0 is already published on npm
🦋  warn @astrojs/svelte is not being published because version 5.5.0 is already published on npm
🦋  warn @astrojs/tailwind is not being published because version 5.1.0 is already published on npm
🦋  info @astrojs/vercel is being published because our local version (0.0.0-refactor-prerender-20240614140807) has not been published on npm
🦋  warn @astrojs/vue is not being published because version 4.4.0 is already published on npm
🦋  warn @astrojs/web-vitals is not being published because version 0.2.1 is already published on npm
🦋  warn @astrojs/internal-helpers is not being published because version 0.4.0 is already published on npm
🦋  warn @astrojs/markdown-remark is not being published because version 5.1.0 is already published on npm
🦋  warn @astrojs/studio is not being published because version 0.1.0 is already published on npm
🦋  warn @astrojs/telemetry is not being published because version 3.1.0 is already published on npm
🦋  warn @astrojs/underscore-redirects is not being published because version 0.3.3 is already published on npm
🦋  warn @astrojs/upgrade is not being published because version 0.3.1 is already published on npm
🦋  info Publishing "astro" at "0.0.0-refactor-prerender-20240614140807"
🦋  info Publishing "@astrojs/vercel" at "0.0.0-refactor-prerender-20240614140807"
🦋  success packages published successfully:
🦋  [email protected]
🦋  @astrojs/[email protected]
🦋  Creating git tags...
🦋  New tag:  [email protected]
🦋  New tag:  @astrojs/[email protected]

Build Log


> [email protected] build /home/runner/work/astro/astro
> turbo run build --filter=astro --filter=create-astro --filter="@astrojs/*" --filter="@benchmark/*"

• Packages in scope: @astrojs/alpinejs, @astrojs/cloudflare, @astrojs/db, @astrojs/internal-helpers, @astrojs/lit, @astrojs/markdoc, @astrojs/markdown-remark, @astrojs/mdx, @astrojs/netlify, @astrojs/node, @astrojs/partytown, @astrojs/preact, @astrojs/prism, @astrojs/react, @astrojs/rss, @astrojs/sitemap, @astrojs/solid-js, @astrojs/studio, @astrojs/svelte, @astrojs/tailwind, @astrojs/telemetry, @astrojs/underscore-redirects, @astrojs/upgrade, @astrojs/vercel, @astrojs/vue, @astrojs/web-vitals, @benchmark/timer, astro, create-astro
• Running build in 29 packages
• Remote caching enabled
::group::@astrojs/telemetry:build
cache hit, suppressing logs 087d8ef0d2c1a399
::endgroup::
::group::@astrojs/internal-helpers:build
cache hit, suppressing logs a304b24d43af2790
::endgroup::
::group::@astrojs/markdown-remark:build
cache hit, suppressing logs c990d2b3884cf01b
::endgroup::
::group::@astrojs/prism:build
cache hit, suppressing logs 61f70c3cc32fdd90
::endgroup::
::group::create-astro:build
cache hit, suppressing logs a79ef3399ce64fd0
::endgroup::
::group::@astrojs/upgrade:build
cache hit, suppressing logs e97cdb908e0f028d
::endgroup::
::group::astro:build
cache miss, executing 35d9ad82218194e6

> [email protected] build /home/runner/work/astro/astro/packages/astro
> pnpm run prebuild && astro-scripts build "src/**/*.{ts,js}" && tsc && pnpm run postbuild


> [email protected] prebuild /home/runner/work/astro/astro/packages/astro
> astro-scripts prebuild --to-string "src/runtime/server/astro-island.ts" "src/runtime/client/{idle,load,media,only,visible}.ts"


> [email protected] postbuild /home/runner/work/astro/astro/packages/astro
> astro-scripts copy "src/**/*.astro" && astro-scripts copy "src/**/*.wasm"

::endgroup::
::group::@astrojs/lit:build
cache miss, executing 6e2b196fab4d4758

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/lit
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/alpinejs:build
cache miss, executing 9b438d196c8ff412

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/alpinejs
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@benchmark/timer:build
cache miss, executing 74d47b1307df11b3

> @benchmark/[email protected] build /home/runner/work/astro/astro/benchmark/packages/timer
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/react:build
cache miss, executing 67fed34848774501

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/react
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/mdx:build
cache miss, executing b59271100633ee75

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/mdx
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/node:build
cache miss, executing b1736a541c87f468

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/node
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/markdoc:build
cache miss, executing 98ee6ea1fcde5d6d

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/markdoc
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/preact:build
cache miss, executing 8b40af4e740c28f5

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/preact
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/svelte:build
cache miss, executing c7485f491f9a9757

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/svelte
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/vue:build
cache miss, executing e5fbb0e14f3da417

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/vue
> astro-scripts build "src/index.ts" && astro-scripts build "src/editor.cts" --force-cjs --no-clean-dist && tsc

::endgroup::
::group::@astrojs/studio:build
cache miss, executing df446708b12951ce

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/studio
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/partytown:build
cache miss, executing 262f354226e34f0e

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/partytown
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/tailwind:build
cache miss, executing 2e124d27be008050

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/tailwind
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/rss:build
cache miss, executing 419f68c6f27f15d9

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/astro-rss
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/solid-js:build
cache miss, executing d4bd0ed0c37a5524

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/solid
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/underscore-redirects:build
cache miss, executing a83e66798593ea3d

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/underscore-redirects
> astro-scripts build "src/**/*.ts" && tsc -p tsconfig.json


> @astrojs/[email protected] postbuild /home/runner/work/astro/astro/packages/underscore-redirects
> astro-scripts copy "src/**/*.js"

::endgroup::
::group::@astrojs/vercel:build
cache miss, executing 6be481cfc951a104

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/vercel
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/sitemap:build
cache miss, executing cb47ecee496b56cf

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/sitemap
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::
::group::@astrojs/db:build
cache miss, executing 95dedfc1e3ba4dfe

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/db
> astro-scripts build "src/**/*.ts" && tsc && pnpm types:virtual


> @astrojs/[email protected] types:virtual /home/runner/work/astro/astro/packages/db
> tsc -p ./tsconfig.virtual.json

::endgroup::
::group::@astrojs/web-vitals:build
cache miss, executing 3515c7ffb364a1ad

> @astrojs/[email protected] build /home/runner/work/astro/astro/packages/integrations/web-vitals
> astro-scripts build "src/**/*.ts" && tsc

::endgroup::

 Tasks:    27 successful, 27 total
Cached:    6 cached, 27 total
  Time:    51.023s

bluwy added 6 commits June 7, 2024 21:39

Refactor code

b0cc443

Merge branch 'main' into refactor-prerender

1e084f0

Fix tests

0e4f0c0

Add test

618cf2d

Format code

570271b

Final fix

471b820

github-actions bot added pkg: integration Related to any renderer integration (scope) pkg: astro Related to the core `astro` package (scope) labels Jun 12, 2024

Create three-boxes-sniff.md

2bc718c

bluwy commented Jun 12, 2024

View reviewed changes

This comment was marked as outdated.

Sign in to view

This was referenced Jun 12, 2024

WIP Add build option to build only ssr #11155

Closed

WIP add ssr prerendering workaround withastro/adapters#270

Closed

bluwy added 2 commits June 13, 2024 23:46

Fix stuff

85b5c02

Merge branch 'main' into refactor-prerender

135a93b

alexanderniebuhr mentioned this pull request Jun 13, 2024

feat(cloudflare): Exclude prerender and unused chunks from server bundle withastro/adapters#222

Merged

Fix node fail

bb31986

ematipico reviewed Jun 14, 2024

View reviewed changes

packages/astro/src/core/build/plugins/plugin-prerender.ts Outdated Show resolved Hide resolved

packages/astro/test/astro-assets-prefix.test.js Show resolved Hide resolved

bluwy added 2 commits June 14, 2024 18:06

Add test tip [skip ci]

59400f7

Remove recursion

d0bd620

ematipico approved these changes Jun 14, 2024

View reviewed changes

matthewp approved these changes Jun 14, 2024

View reviewed changes

bluwy merged commit e22be22 into main Jun 17, 2024
14 checks passed

bluwy deleted the refactor-prerender branch June 17, 2024 06:23

bluwy mentioned this pull request Jun 17, 2024

test for bug reproduction #10738

Closed

astrobot-houston mentioned this pull request Jun 17, 2024

[ci] release #11233

Merged

bluwy mentioned this pull request Jun 17, 2024

Several chunk files are added to the SSR bundle for every prerendered mardkown files #10552

Closed

1 task

This was referenced Jul 1, 2024

4.11 not pruning satori build chunks, increasing build sizes #11377

Closed

Fix prerendering with unused dynamic chunks #11387

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Refactor prerendering chunk handling #11245

Refactor prerendering chunk handling #11245

bluwy commented Jun 12, 2024

changeset-bot bot commented Jun 12, 2024 •

edited

Loading

bluwy left a comment

This comment was marked as outdated.

This comment was marked as outdated.

ematipico left a comment

bluwy commented Jun 14, 2024

github-actions bot commented Jun 14, 2024

Refactor prerendering chunk handling #11245

Refactor prerendering chunk handling #11245

Conversation

bluwy commented Jun 12, 2024

Changes

Testing

Docs

changeset-bot bot commented Jun 12, 2024 • edited Loading

🦋 Changeset detected

bluwy left a comment

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

ematipico left a comment

Choose a reason for hiding this comment

bluwy commented Jun 14, 2024

github-actions bot commented Jun 14, 2024

changeset-bot bot commented Jun 12, 2024 •

edited

Loading