Update docs on how to run Pants in CI. #16503
Conversation
Explain where remote caching can come in handy, and why.
Co-authored-by: Eric Arellano <[email protected]>
|
PTAL - I had to revamp a lot once I dug deeper into GHA caching and its immutability. |
|
|
||
| - `$HOME/.cache/pants/setup`<br> | ||
| This is the Pants bootstrap directory. Cache this against the version, as specified | ||
| in `pants.toml`. E.g., if using GitHub Actions you can use: |
There was a problem hiding this comment.
The named caches are harder, but for this it seems we could provide a step or two that used python + toml to parse out the version and set it as an output for the cache step to use as the right most key instead. In fact, we could can that as an action in pants build/actions and just point folks to use that in this setup rec. For later.
There was a problem hiding this comment.
You beat me to it: pantsbuild/example-python#104 (comment)
There was a problem hiding this comment.
Maybe, but let's get this in for now, as it's a big improvement on what we had before, and we're not trying to be everyone's devops people, just give a sense of what can profitably be cached.
| path: | | ||
| ~/.cache/pants/setup | ||
| key: pants-setup-${{ runner.os }}-${{ hashFiles('pants.toml') }} | ||
| restore-keys: | |
There was a problem hiding this comment.
This comment is left as a marker for all restore-key related comments which I won't repeat from pantsbuild/example-python#104 (comment)
There was a problem hiding this comment.
(same comment about not being comfortable with using restore-keys)
| If you're not using a fine-grained [remote caching](doc:remote-caching-execution) service, | ||
| then you may also want to preserve the local Pants cache at `$HOME/.cache/pants/lmdb_store`. | ||
| This has to be invalidated on any file that can affect any process, e.g., `hashFiles('**/*')`. | ||
| Computing such a coarse hash, and saving and restoring large directories, can be unwieldy. | ||
| So this may be impractical and slow on medium and large repos. | ||
| A [remote cache service](doc:remote-caching-execution) integrates with Pants's fine-grained | ||
| invalidation and avoids these problems, and is recommended for the best CI performance. |
There was a problem hiding this comment.
Another way to skin this cat that I don't see being discussed it self-hosted GitHub Runners, which can leverage a persistent filesystem.
https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners
There was a problem hiding this comment.
I have zero experience doing that, and again, it is out of scope for this change I think. We couldn't document that until we tried it.
There was a problem hiding this comment.
I'll file an issue and we can document later
There was a problem hiding this comment.
Is this something you've actually done? It's a really interesting approach if so!
|
This now links to example-python for specifics, rather than copying them here. |
- More details on how to use the CI cache - Explain where remote caching can come in handy, and why.
- More details on how to use the CI cache - Explain where remote caching can come in handy, and why.
Explain where remote caching can come in handy, and why.