Website: Improve storage for static assets

[swissspidy]

Task Description

This is a long-standing open to-do item related to our static assets hosting. With more templates in the making, the repo size will grow considerably, so we need a solution here.

We can keep the microsite and small images in static-site as today, but the template assets we should move at some point.

In other words:

There will be 50 templates that we need to host somewhere and make available via some API.

Some thoughts:

• Move assets to separate repository
  • This will just move the problem and make another repository very big
• Use Git LFS
  • Comes with storage & bandwidth quotas
  • makes the repo less portable
  • is more meant for large files, not many files
• Move everything to GCP bucket / Firebase Storage
  • Less easy to contribute

---

Current git-sizer output

$ git-sizer
Processing blobs: 48401                        
Processing trees: 57445                        
Processing commits: 12597                        
Matching commits to trees: 12597                        
Processing annotated tags: 0                        
Processing references: 121                        
| Name                         | Value     | Level of concern               |
| ---------------------------- | --------- | ------------------------------ |
| Overall repository size      |           |                                |
| * Blobs                      |           |                                |
|   * Total size               |  13.8 GiB | *                              |
|                              |           |                                |
| Biggest objects              |           |                                |
| * Commits                    |           |                                |
|   * Maximum size         [1] |  60.2 KiB | *                              |
| * Blobs                      |           |                                |
|   * Maximum size         [2] |  34.8 MiB | ***                            |
|                              |           |                                |
| Biggest checkouts            |           |                                |
| * Maximum path depth     [3] |    11     | *                              |
| * Maximum path length    [4] |   143 B   | *                              |
| * Total size of files    [5] |  1.19 GiB | *                              |

[1]  8616cbca8b04dba66f16c85dc84204947b78c430
[2]  7bbf0ee5cae3d6f49ddb4745b8734d0af2131c49 (refs/heads/static-site:public/video/NoDaysCycle-v1.mp4)
[3]  01b18435bb1897cb1cd522581171f197884f0ba0 (9f715490f21f1ef139194ca7afdabe815e5a6167^{tree})
[4]  b4fbe6fae6c474162502a14798175be21af8cd5b (468501b165a6249b270947f64c04ffa18720dc56^{tree})
[5]  706025ec537f5027d35b45e12c7015f7a3661e83 (d3cf04a00c3b73b0fd3ef575a47d964990650d78^{tree})

[swissspidy]

@choumx would love to hear your thoughts on that

[dreamofabear]

I don't have experience with it but Git LFS sounds promising due to git versioning and GitHub support. It should be very inexpensive for our expected usage.

[swissspidy]

I'm a bit worried about the steps required to turn off LFS again if needed, but we can give it a try.

No matter what option we choose, the current repo size will probably remain the same since git history is left untouched.

[dreamofabear]

current repo size will probably remain the same

We could purge history for the template assets. It might speed up some of our CI steps.

[swissspidy]

current repo size will probably remain the same

We could purge history for the template assets. It might speed up some of our CI steps.

Checking out the repo in CI only takes a couple of seconds, don't think we can speed that up more.

Any improvements due to purging history would mostly benefit contributors cloning the repo. Right now new contributors will clone 1GB worth of files even when never using the static-site branch.

[swissspidy]

Documenting the steps required to set up Git LFS (WIP):

1. Install Git LFS (e.g. via brew install git-lfs or by following https://docs.github.com/en/github/managing-large-files/installing-git-large-file-storage)
2. Run git lfs install --manual
   This avoid conflicts with Husky's Git hooks (see Git LFS and husky prepush hook? typicode/husky#108 and Support merging with git LFS hooks typicode/husky#729), but requires manually updating the Git hooks in .git/hooks

And here's what I've been doing to migrate the static-site branch:

1. Check out static-site branch
2. Run the following command:

   git lfs track "./public/static/**/*.png"
   git lfs track "./public/static/**/*.jpg"
   git lfs track "./public/static/**/*.mp4"
   git lfs track "./public/static/**/*.webm"

   This will update the .gitattributes file and track existing + new files in Git LFS.
3. Dry run to see object sizes:

    git lfs migrate info static-site --include="public/static/**/*.png,public/static/**/*.jpg,public/static/**/*.mp4,public/static/**/*.webm"

4. Run actual migration:

   git lfs migrate import static-site --include="public/static/**/*.png,public/static/**/*.jpg,public/static/**/*.mp4,public/static/**/*.webm"

   ⚠️ This will rewrite history for the static-site branch (but not others)

All contributors modifying assets in static-site will have to install Git LFS, see https://docs.github.com/en/github/managing-large-files/collaboration-with-git-large-file-storage

Note:

I am not updating the website images+videos to use LFS with this (yet). But it could be done:

git lfs track "./public/img/*.png"
git lfs track "./public/img/**/*.png"
git lfs track "./public/img/*.jpg"
git lfs track "./public/img/**/*.jpg"
git lfs track "./public/video/*.mp4"
git lfs track "./public/video/**/*.mp4"
git lfs track "./public/video/*.webm"
git lfs track "./public/video/**/*.webm"

git lfs migrate info static-site --include="public/img/*.png,public/img/**/*.png,public/img/*.jpg,public/img/**/*.jpg,public/video/*.mp4,public/video/**/*.mp4,public/video/*.webm,public/video/**/*.webm"

git lfs migrate import static-site --include="public/img/*.png,public/img/**/*.png,public/img/*.jpg,public/img/**/*.jpg,public/video/*.mp4,public/video/**/*.mp4,public/video/*.webm,public/video/**/*.webm"

[swissspidy]

Closing for now

[szepeviktor]

Please be very brave and remove those files with their commit history breaking all forks but
SAVING VERY LONG MINUTES for everybody for the foreseeable future, causing joy 😂

[szepeviktor]

Downloading 1.17 GB of data makes contributors go away.