S3 Rebuild Caching #1150
Closed
S3 Rebuild Caching #1150
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Co-authored-by: Pavel Grinchenko <[email protected]>
Co-authored-by: Pavel Grinchenko <[email protected]>
There was double remote name, so it didn't work.
fix synopsis sections of list, update cmd refs
* april draft * add new DVC features * revisions * comments link * add link in new release
* readme/contributing: update after migration to Gatsby * README: fix copyright, include proper links, refactor sections * docs contib guide: update, fix linters a litle bit * contrib: update and improve writing a blog post section * readme/contrib: address PRs review * readme/contrib: address second review
rogermparent
commented
Apr 15, 2020
scripts/deploy-with-s3.js
Outdated
| const publicDirName = 'public' | ||
| const publicDirEntry = [publicDirName, '/'] | ||
| const cacheDirEntry = ['.cache', '-cache/'] |
There was a problem hiding this comment.
This tuple syntax can be changed to something like objects if we're not all into it.
…main (iterative#1151) Co-authored-by: Pavel Grinchenko <[email protected]>
…iterative#1154) already replace original href with redirect if it exists Co-authored-by: Pavel Grinchenko <[email protected]>
This way the code doesn't depend on the source file's relative directory.
I'm not sure if there's any rate limits in the stack that will cause this to break, but on first blush this looks like a quick and easy improvement. The cached folders are also now accessed by a shared array that is hard-coded for now. This both makes them easier to work with uniformly and open to sourcing directories from elsewhere in the future.
I believe the crux of what keeps this PR from working is that `public` and `.cache` the folders were being uploaded to `public` and `,cache` the S3 prefixes when the site is deployed from the root prefix. This change alters how the cache directories are stored: now as tuple pairs with the local directory in slot 0 and a string that's appended to the base prefix in slot 2. `public` work with the root prefix and the `.cache` folder works with the root prefix with `-cache` appended to it such that it's another "folder" that doesn't get downloaded with the root prefix. I find the tuple syntax more concise, but I'm open to changing it if we prefer objects for better grokking at a glance. This change also moves the responsibility of cache culling into an `onPostBuild` Gatsby hook. This means we don't have to parse the sitemap XML after the fact, and as such the XML parser used to do it is removed. Since this replaces `cleanUpPageData`, it also invalidates the whole `page-data-utils` file which has been removed.
rogermparent
force-pushed
the
rebuilds-caching
branch
from
2fc42b8 to
f1892f6
Compare
shcheklein
approved these changes
Apr 16, 2020
rogermparent
force-pushed
the
rebuilds-caching
branch
from
7f7120f to
f1892f6
Compare
|
I rebased my fork's branch onto master earlier today without thinking of the ramifications on the PR. I think my changes will have to be opened up on a new PR to master. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is another pretty fat commit, and with less of an excuse than the previous one. However, this one trades theoretical messiness for functionality!
The previous solution would upload public to the prefix
${s3Prefix}/public/, but I believe the website is served from${s3Prefix}. This solution fixes that and changes.cacheto go into the prefix${s3Prefix}-cacheso it doesn't get picked up in "folder" queries for the root prefix.Also, the pruning functionality that used to happen in the deploy script by parsing the XML sitemap is now moved to a more "native" solution in Gatsby's
onPostBuildhook.I've tested this locally and successfully got rebuilds with no image processing after running
gatsby cleanand then downloaded a previous cache with the deploy script. (It even handled a partial cache after I unthrottled the page count!)