e2e_regression: Manage RPC cache with git-lfs

[mitjat]

This PR removes all the large RPC-cache files from the regular git repo, and re-adds them as git-lfs-tracked files. This means that the files in the repo will contain just a textual pointer to a LFS server, which will store the actual data.

I did not move the RPC caches to git-lfs for old commits. That would require rewriting our entire history.

Storage of LSF-tracked files is on GitHub. My understanding is that LFS-managed files also keep counting against the same storage quota for our repo. The main advantages that I see are:

• When cloning the repo, the contents of the LFS server are downloaded only for the currently-checked-out version, on-demand. Makes clones faster. However, it also makes offline commiting and checkouts impossible? I tested, offline works fine, git-lfs keeps a local copy of un-uploaded changes.
• If we ever want to drop old versions of the e2e_regression RPC cache, we don't need to rewrite history (as we'd have to do now); instead, we should be able to just drop the backing files on the LFS server and keep the pointers in the actual repo unchanged (but broken). That said, it looks like this kind of server-side LFS pruning is not standardized and not currently supported by github 😯.

Note to developers: To use LFS-managed files (i.e. to run e2e_regression locally), you have to have git-lfs installed.

[Andrew7234]

Fwiw I haven't had noticable pain points from the cache file size. Also, git-lfs knows to automatically fetch the files from github? or do we have to configure that in our local env?

 du -sh ./*                                                            andrewlow@Beleriand
 38M	./consensus
1.9M	./emerald

[Andrew7234]

should we check in the .pmt files so that everything in rpc-cache is in lfs?

[mitjat]

🤷 I don't see a big benefit either way. This felt a tiny bit more future proof in case we store any "normal" files under rpc-cache/, e.g. a README.

[pro-wh]

oh we only need the .psg files right? pogreb scans them and generates the rest. could save time to deliver these pix files. what's pmt?

[mitjat]

Fwiw I haven't had noticable pain points from the cache file size.

The cache itself is not huge. But the repo stores the entire history, and every time the cache contents change just a tiny bit, git stores a whole new 38MB blob. Times ~20 versions of the cache is ... a lot. And we're considering adding more e2e regressions (against multiple history ranges).

Also, git-lfs knows to automatically fetch the files from github? or do we have to configure that in our local env?

It's all automatic and transparent, but you do have to install git-lfs first. And then maybe you have to call git lfs install in the repo? Instructions talk about it, but I never did it, and things still worked smoothly.

[pro-wh]

this is it, an excuse to install git-lfs