-
Notifications
You must be signed in to change notification settings - Fork 8.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ts/build-refs] implement experimental remote cache #91012
[ts/build-refs] implement experimental remote cache #91012
Conversation
df52893
to
0bf7bc1
Compare
d81599a
to
9d0985d
Compare
Pinging @elastic/kibana-operations (Team:Operations) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@spalger just two things:
1 - are we planning to enabled the use of the remote cache by default?
2 - do we have garbage collection enabled in the remote google cloud storage bucket? I think probably we can auto delete archives older than 30 days.
Is it possible that your local cache is more up-to-date than the cache being requested? (example: running back-to-back bootstraps) |
That's a fantastic point. Near the end of my testing I switched from writing the |
If it turns out that this works great than absolutely
I'm open to turning it on, but for now I left it off so that people checking out old branches or commits would still be able to benefit from the cache. |
@elasticmachine merge upstream |
@tylersmalley this is not a scenario that would wipe out the |
👍 I am good moving forward with this change so we can do more testing when the cache is being populated and we can selectively opt-in. |
Pinging @elastic/fleet (Team:Fleet) |
💚 Build SucceededMetrics [docs]
History
To update your PR or re-run it, just comment with: |
* [ts/build-refs] implement experimental remote cache * delete old tests * add some more tests * add some docs and a readme * fix kibanaPackageJson usage Co-authored-by: spalger <[email protected]> Co-authored-by: Kibana Machine <[email protected]>
Backport result
|
…91299) Co-authored-by: spalger <[email protected]> Co-authored-by: Kibana Machine <[email protected]> Co-authored-by: Spencer <[email protected]>
RE #90281
In order to decrease the amount of time people have to spend bootstrapping I'd like to experiment with storing a cache of all the
outDir
s of each ts project referenced fromtsconfig.refs.json
. This PR implements the logic to build, upload, and download those caches.This cache reduces the execution time of
node scripts/build_ts_refs
in a clean repo to ~40 seconds on my machine when a cache is downloaded fromhttps://ts-refs-cache.kibana.dev
and there are minimal changes to local files.See https://github.com/spalger/kibana/blob/implement/distributed-ts-build-cache/src/dev/typescript/ref_output_cache/README.md for an explanation about how the cache logic works.
To build the cache we zip up each
outDir
, and then zip up those zips, making it simple to ship the cache around, pretty light weight, easier to keep a backlog of caches and clean them up, cheaper to extract a portion of the cache, and makes extraction parallelizable. I've experimented with tarballs and a single zip, but the speed of extracting a single large tarball/zip was quite slow.Config options:
BUILD_TS_REFS_CACHE_ENABLE=true
ornode scripts/build_ts_refs --cache
will enable the experimental cache, the cache is not enabled by default.BUILD_TS_REFS_CACHE_CAPTURE=true
will create the cache zip intarget/ts_refs_cache
Successful test of building and uploading cache as defined in the baseline job: https://kibana-ci.elastic.co/job/elastic+kibana+pipeline-pull-request/105834/