-
Notifications
You must be signed in to change notification settings - Fork 19
Rewrite Static Asset References? #224
Comments
we could revisit the static asset rewrite in parcel. but this would result in a 2 stage build process again, where the referenced assets are renamed and copied to some Dist folder. |
I might have an idea how to solve this, but it involves a bit of Fastly magic:
The For extra credit:
I think the HTML/CSS/JS parser infrastructure we have available is robust enough to enable this. |
…si to replace a static URL with a cacheable static URL and .hlx_[0-9a-f]+ for resources that are cached forever in the browser In order to increase the cache efficiency of static resources, we are implementing a multi-stage process which replaces the generic, short and uncacheable URLs of static resources like `/scripts/site.js` with long and version-specific URLs like `/scripts/site.js.hlx_f7c3bc1d808e04732adf679965ccc34ca7ae3441`. This is done by offering a virtual resource like `/scripts/site.js.esi` that can be included in an ESI tag like `<script src="<esi:include src="/scripts/site.js.esi" />">` from within HTML, JS or CSS (the replacement of `.js` and `.css` file endings iwth `.js.esi` and `.css.esi` will be handled in helix-pipeline and helix-static at a later point in time). This virtual resource will then simply replace the short URL `/scripts/site.js` with the long-cachable URL `/scripts/site.js.hlx_f7c3bc1d808e04732adf679965ccc34ca7ae3441` through the ESI resolution process. Any static resource requested with a long-cachable URL will be delivered with Cache-Control headers setting expiry 366 days into the future. Partial implementation of adobe/helix-pipeline#224 (comment)
somehow, I don't like that we have to do a separate request to static, just to figure out a hash, that we already know during build time. the entire static handling (htdocs) is somewhat inconsistent anyways, since the state effective during the deployment, can change when someone updates the files in the htdocs repository later. so maybe would be better to update the static url with the commit sha of the static repository (similar to the
what happens, if the etag doesn't match the etag responded from the actual content? i.e. if the content changed in the meantime? |
Same as 396c55b but for JavaScript. For now, only `import` statements are affected depends on adobe/helix-publish#61 and contributes to adobe/helix-pipeline#224 (comment)
@tripodsan the POC I have in progress at adobe/helix-publish#61 and https://github.com/adobe/helix-cli/tree/static-esi solves this a bit more elegantly by taking step 2 and 3
and implementing it entirely in VCL, i.e. Fastly will make the request to get the ETag. The disadvantage of your approach is that it requires re-deployment whereas mine allows the static content to evolve separately from the normal content and code. I'd explain more, but I have to help retrieve an (actual) Easter egg that we've hidden too well. |
yes. I understand. but it requires 2 requests for the static.
I don't think this a disadvantage. on the contrary. I think that static content should be frozen at the time of deployment. since scripts and style should be treated like code, hence have a very strong relationship to the templates. |
@filmaj @stevengill WDYT? should the static content (especially javavscripts and css) be tied to a deployment or is it useful to be able to update it independently? I think for production, it is crucial, that the scripts and styles match the HTML created by the templates, thus are tied to them. I think for the static site deployments, people usually bundle those inside the webpack (or parcels). |
@tripodsan tricky question. For me as a developer of the devsite, I am OK with deploying every time I make a static asset update to my site. However, I don't think that is acceptable for the distributed nature of the future devsite content. That is, for e.g. product teams building their own product micro and docs sites on top of our site, I wouldn't want a static asset change in one of their content repos to be gated by a devsite deploy. Just to be clear, for a strain like this in the devsite:
I do not want to have to do a |
Wait a second, I am confusing Since static files are only managed in the devsite, which I control, I am OK with requiring a deploy/publish for updating them. However, if static file control would at any point slip out of my control, I would not be OK with that. |
That's the worst-case scenario. The best-case scenario is no requests at all because the asset is cached in the browser (with confidence it won't change). The only other way I can see of achieving that would be to use a bundler like Webpack or Parcel for all client assets, which would rewrite the entire CSS and JS every time, forcing you not just to have compile targets committed back to git (which I find distasteful) but also to break the history of these compile targets. Fastly making two requests over one is not a concern for me as Fastly is very efficient at making requests. Invoking runtime twice would be more of a concern. |
the other idea would be to pass the |
but initially, it will make 2. first the HEAD request, and then the GET request. |
That's almost what I'd be doing except:
With my ESI-based approach we can still switch the implementation of the
|
hmm...ok. I see. the esi is more powerful :-) so:
this would be done in the
if possible, this should be done in fastly, by bypassing the
this is similar to (1), just that it uses the there is also
|
I think we can implement all three options in Fastly.
|
but we update the add a new strains:
- name: default
content: ...
static: ...
code: ....
heads:
- url: https://github.com/adobe/project-helix.io.git#master
sha: 879237844530beea65db3434a1074f66R397
- url: https://github.com/adobe/project-helix-content.io.git#stage
sha: ... |
@trieloff how are you going to decide, which strategy to use? a) use whatever the current branch points to (deployment independent version) |
# [1.1.0](v1.0.0...v1.1.0) (2019-04-29) ### Bug Fixes * **static:** normalize URLs in rewritten CSS and JS ([25964ba](25964ba)) * **static:** turn on ESI processing when ESI flag is set ([8b2436b](8b2436b)) ### Features * **static:** add ESI aliasing support for JavaScript modules ([1fa564d](1fa564d)), closes [/github.com/adobe/helix-pipeline/issues/224#issuecomment-476690621](https://github.com//github.com/adobe/helix-pipeline/issues/224/issues/issuecomment-476690621) * **static:** Rewrite CSS URLs to Static ESI URLs so that better caching can be achieved ([396c55b](396c55b)), closes [adobe/helix-publish#61](adobe/helix-publish#61) [adobe/helix-pipeline#267](adobe/helix-pipeline#267)
# [1.1.0](v1.0.0...v1.1.0) (2019-04-29) ### Bug Fixes * **package:** update @adobe/helix-shared to version 0.10.5 ([8efda26](8efda26)) * **static:** normalize URLs in rewritten CSS and JS ([25964ba](25964ba)) * **static:** turn on ESI processing when ESI flag is set ([8b2436b](8b2436b)) ### Features * **static:** add ESI aliasing support for JavaScript modules ([1fa564d](1fa564d)), closes [/github.com/adobe/helix-pipeline/issues/224#issuecomment-476690621](https://github.com//github.com/adobe/helix-pipeline/issues/224/issues/issuecomment-476690621) * **static:** Rewrite CSS URLs to Static ESI URLs so that better caching can be achieved ([396c55b](396c55b)), closes [adobe/helix-publish#61](adobe/helix-publish#61) [adobe/helix-pipeline#267](adobe/helix-pipeline#267)
# [1.10.0](v1.9.2...v1.10.0) (2019-04-30) ### Bug Fixes * **package:** update @adobe/helix-shared to version 0.10.5 ([54709a9](54709a9)) * **static:** change extension from .esi to .esi ([a936918](a936918)) ### Features * **html:** enable HAST parsing and serialization of HTML responses ([224c665](224c665)), closes [#285](#285) * **html:** rewrite relative asset references to ESI resources ([dce696e](dce696e)), closes [#267](#267) * **html:** rewrite static asset references to ESI includes that provide stable URLs ([aa2538f](aa2538f)), closes [#224](#224)
🎉 This issue has been resolved in version 1.10.0 🎉 The release is available on: Your semantic-release bot 📦🚀 |
# [2.0.0](v1.0.0...v2.0.0) (2019-05-06) ### Bug Fixes * **package:** update @adobe/helix-shared to version 0.10.5 ([8efda26](8efda26)) * **package:** update @adobe/helix-shared to version 0.11.0 ([0de812b](0de812b)) * **package:** update @adobe/helix-shared to version 1.1.0 ([fabb0ab](fabb0ab)) * **package:** update chokidar to version 3.0.0 ([4688a0f](4688a0f)) * **package:** update dotenv to version 8.0.0 ([0786a94](0786a94)) * **package:** update snyk to version 1.154.1 ([3038ec9](3038ec9)) * **package:** update snyk to version 1.155.0 ([a8bb6d6](a8bb6d6)) * **package:** update snyk to version 1.156.0 ([e321706](e321706)) * **package:** update snyk to version 1.159.0 ([adeb477](adeb477)) * **package:** update snyk to version 1.161.0 ([79be958](79be958)) * **package:** update snyk to version 1.161.1 ([72ad988](72ad988)) * **static:** always return entry path for 404 errors ([84fe876](84fe876)) * **static:** cache error responses for 5 minutes ([4247842](4247842)) * **static:** handle more 404 errors with path ([ed5a887](ed5a887)) * **static:** normalize URLs in rewritten CSS and JS ([25964ba](25964ba)) * **static:** prevent possible XSS by sanitizing output ([97d6387](97d6387)) * **static:** return the original URL in case a static resource cannot get retieved for an ESI include ([3f6b3ff](3f6b3ff)), closes [#813](#813) * **static:** turn on ESI processing when ESI flag is set ([8b2436b](8b2436b)) * **static:** use .url instead of .esi as extension for immutable resources ([70b9674](70b9674)) ### Documentation * **changelog:** explain switchover to version 2.0 ([920b39c](920b39c)) ### Features * **static:** add ESI aliasing support for JavaScript modules ([1fa564d](1fa564d)), closes [/github.com/adobe/helix-pipeline/issues/224#issuecomment-476690621](https://github.com//github.com/adobe/helix-pipeline/issues/224/issues/issuecomment-476690621) * **static:** Rewrite CSS URLs to Static ESI URLs so that better caching can be achieved ([396c55b](396c55b)), closes [adobe/helix-publish#61](adobe/helix-publish#61) [adobe/helix-pipeline#267](adobe/helix-pipeline#267) ### BREAKING CHANGES * **changelog:** not breaking any functionality, bumping to version 2.0 as the v1.* name space for git tags has already been polluted
Same as 396c55bf462a2dadd1897a9f05c370b9a6dbfb18 but for JavaScript. For now, only `import` statements are affected depends on adobe/helix-publish#61 and contributes to adobe/helix-pipeline#224 (comment)
# 1.0.0 (2019-05-14) ### Bug Fixes * **static:** always return entry path for 404 errors ([552ef3d](552ef3d)) * **static:** cache error responses for 5 minutes ([3173cea](3173cea)) * **static:** handle more 404 errors with path ([2c1ac18](2c1ac18)) * **static:** normalize URLs in rewritten CSS and JS ([f238adb](f238adb)) * **static:** prevent possible XSS by sanitizing output ([0795581](0795581)) * **static:** return the original URL in case a static resource cannot get retieved for an ESI include ([453c724](453c724)), closes [#813](https://github.com/adobe/helix-static/issues/813) * **static:** turn on ESI processing when ESI flag is set ([a6d96b8](a6d96b8)) * **static:** use .url instead of .esi as extension for immutable resources ([d6c89f8](d6c89f8)) ### Features * **pipeline:** consistently use `context` instead of payload. ([ddea2cb](ddea2cb)), closes [#743](#743) * **static:** add ESI aliasing support for JavaScript modules ([6c0d0d5](6c0d0d5)), closes [/github.com/adobe/helix-pipeline/issues/224#issuecomment-476690621](https://github.com//github.com/adobe/helix-pipeline/issues/224/issues/issuecomment-476690621) * **static:** Rewrite CSS URLs to Static ESI URLs so that better caching can be achieved ([75b47e5](75b47e5)), closes [adobe/helix-publish#61](adobe/helix-publish#61) [adobe/helix-pipeline#267](adobe/helix-pipeline#267)
# 1.0.0 (2019-06-24) ### Bug Fixes * **static:** process ESI for static resources, too ([8ea480d](8ea480d)) * .snyk & package.json to reduce vulnerabilities ([d68bf41](d68bf41)) * .snyk & package.json to reduce vulnerabilities ([c2314e5](c2314e5)) * **deploy:** use nodejs:10 container ([b64541e](b64541e)), closes [#55](#55) * **dictionaries:** set missing directory index ([ef56581](ef56581)), closes [#59](#59) * **linting:** linting again... ([b496576](b496576)) * **linting:** linting... ([275494f](275494f)) * **monitoring:** remove epsagon token from source code ([be35e31](be35e31)) * **package:** update @adobe/fastly-native-promises to version 1.10.0 ([258c5eb](258c5eb)) * **package:** update @adobe/fastly-native-promises to version 1.4.0 ([c29ed05](c29ed05)) * **package:** update @adobe/fastly-native-promises to version 1.6.0 ([dba6bd0](dba6bd0)), closes [#17](#17) * **package:** update @adobe/fastly-native-promises to version 1.6.1 ([8b755ae](8b755ae)) * **package:** update @adobe/fastly-native-promises to version 1.7.0 ([8955790](8955790)) * **package:** update @adobe/fastly-native-promises to version 1.8.0 ([23c37eb](23c37eb)) * **package:** update @adobe/fastly-native-promises to version 1.9.0 ([68c16f6](68c16f6)) * **package:** update @adobe/fastly-native-promises to version 1.9.1 ([de465f3](de465f3)) * **package:** update @adobe/helix-shared to version 0.10.3 ([7c9caef](7c9caef)) * **package:** update @adobe/helix-shared to version 0.10.4 ([d4b5e92](d4b5e92)) * **package:** update @adobe/helix-shared to version 0.10.5 ([fd67213](fd67213)) * **package:** update @adobe/helix-shared to version 0.11.0 ([d976bcb](d976bcb)) * **package:** update @adobe/helix-shared to version 0.6.2 ([20703f3](20703f3)) * **package:** update @adobe/helix-shared to version 0.7.0 ([d49ba13](d49ba13)) * **package:** update @adobe/helix-shared to version 0.8.0 ([4731732](4731732)) * **package:** update @adobe/helix-shared to version 0.8.1 ([024f137](024f137)) * **package:** update @adobe/helix-shared to version 0.8.3 ([6dd418b](6dd418b)) * **package:** update @adobe/helix-shared to version 0.8.4 ([feaf8eb](feaf8eb)) * **package:** update @adobe/helix-shared to version 1.0.1 ([ffad201](ffad201)), closes [#67](#67) * **package:** update @adobe/helix-shared to version 1.0.2 ([f3c27b6](f3c27b6)) * **package:** update @adobe/helix-shared to version 1.1.1 ([c1bf275](c1bf275)) * **package:** update @adobe/helix-shared to version 1.2.0 ([41faf50](41faf50)) * **package:** update @adobe/helix-shared to version 1.3.1 ([d002710](d002710)) * **package:** update @adobe/helix-shared to version 1.5.0 ([1a61355](1a61355)) * **package:** update fs-extra to version 8.0.1 ([cffa67c](cffa67c)) * **package:** update glob-to-regexp to version 0.4.1 ([5b0782f](5b0782f)) * **proxy:** Don't attempt to recover from 404 errors in proxy strains ([8b84690](8b84690)), closes [#75](#75) * **proxy:** prevent duplicate creation of backends for proxy strains ([c157e8b](c157e8b)) * **proxy:** use FastlyJSON when talking to Fastly API ([9d40b93](9d40b93)) * **static:** add missing slash to GitHub raw URL ([c07f401](c07f401)) * **static:** disable GZip for all backend requests that could contain ESI ([393ae0a](393ae0a)) * **static:** don't redeclare var ([3f23556](3f23556)) * **static:** drop accept-encoding to enforce esi ([2729f64](2729f64)) * **static:** enable compression for ESI-processed resources ([0eaf027](0eaf027)) * **static:** fix syntax error ([852969a](852969a)) * **static:** fix syntax error ([82949b0](82949b0)) * **static:** fix syntax error ([d22c465](d22c465)) * **static:** fix variable name ([cbb125e](cbb125e)) * **static:** fix VCL syntax error ([db4c524](db4c524)) * **static:** fix VCL syntax error ([76749d6](76749d6)) * **static:** handle immutable requests without restart ([79c0ba7](79c0ba7)) * **static:** move esi handling into fetch ([46bc081](46bc081)) * **static:** never use GZip when talking with Runtime, as there might be ESI ([21ce4a1](21ce4a1)) * **static:** no slash between ref and path in static retrieval ([209cd7f](209cd7f)) * **static:** protect against cache poisoning by verifying the etag ([1bc1e24](1bc1e24)) * **static:** remove brackets in URL ([4009ff5](4009ff5)) * **static:** restrict regex for immutable resources ([c71043f](c71043f)) * **static:** strip 302 from URL when requesting a redirect ([fd5a5db](fd5a5db)) * **static:** submit location headers for redirects, making actual redirects possible ([615bf33](615bf33)) * **static:** throw error for invalid resource paths ([92d4c82](92d4c82)) * **static:** turn on ESI processing when backend response requests it ([5459d30](5459d30)) * **static:** unset Accept-Encoding more aggressively ([df42f20](df42f20)) * **static:** use beresp instead of resp for ETag value ([112796d](112796d)) * **static:** use correct location header ([5d1f8c7](5d1f8c7)) * **static:** use correct path for entry for immutable resources ([c66f15f](c66f15f)) * **static:** use local var for new extension ([c995533](c995533)) * **static:** use re instead of req ([245b583](245b583)) * **static:** use req for tracing instead of resp ([806d48d](806d48d)) * **vcl:** bum version ([73d3f8e](73d3f8e)) * **vcl:** copyright & unrelated change ([e88ed93](e88ed93)) * **vcl:** include must be outside of a subroutine ([cef774f](cef774f)) * **vcl:** is line break required ? ([67080b2](67080b2)) * **vcl:** is linebreak required? ([9cc1ea2](9cc1ea2)) * **vcl:** remove unnecessary line breaks ([8502341](8502341)) * **vcl:** respect remapped url also in static ([f404c4d](f404c4d)), closes [#101](#101) ### Features * **monitoring:** add epsagon wrapper ([2d52627](2d52627)), closes [#104](#104) * **proxy:** make proxy strains respect the URL path and backend path ([ff77ba7](ff77ba7)) * **static:** enable asset statification for assets loaded from static URLs ([befc1e2](befc1e2)) * **static:** provide two new resource types for static resources: .esi to replace a static URL with a cacheable static URL and .hlx_[0-9a-f]+ for resources that are cached forever in the browser ([9f8e52c](9f8e52c)), closes [/github.com/adobe/helix-pipeline/issues/224#issuecomment-476690621](https://github.com//github.com/adobe/helix-pipeline/issues/224/issues/issuecomment-476690621) * **static:** rename static-esi to static-url, introduce static-302 which returns a redirect ([403ae1b](403ae1b)), closes [/github.com//pull/61#issuecomment-487664612](https://github.com//github.com/adobe/helix-publish/pull/61/issues/issuecomment-487664612) * **vcl:** Add extensions.vcl ([ae1b52c](ae1b52c)) * **vcl:** add more extension points ([979cfbe](979cfbe)) * **vcl:** Add more extension points ([3dee07e](3dee07e)) * **vcl:** Documentation ([902f6f3](902f6f3)) * **vcl:** Support Custom VCL ([b2ca735](b2ca735)) * **vcl:** testing extension logic ([717c192](717c192)) * **vcl:** Tests for synthetize method ([71bcd2d](71bcd2d)) ### Performance Improvements * **static:** declare static files to be immutable ([f0369b7](f0369b7)) ### BREAKING CHANGES * **proxy:** changes root paths of all proxy strains - fixes #64
Triggered by a conversation here https://github.com/adobe/developer.adobe.com-planning/issues/137#issuecomment-476544754 about optimizing static asset performance, I've been wondering if we should employ tactics to introduce means to create cache-building URLs for static assets.
Problem:
How could cache-building (the opposite of cache-busting) URLs look like?
Instead of having HTML that looks like this:
Adjust the pipeline, so that every static asset will be referred to with an additional cache-key decoration:
b3b7229 would be the same as the current package name. In
helix-static
we'd strip out the cache-decorator from the URL, but remember to set generousCache-Control
headers that encourage long caching.Downsides:
The text was updated successfully, but these errors were encountered: