Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add support for cache eviction policies that can maintain a maximum CI cache size, such as LRU #18294

Open
chrisjrn opened this issue Feb 17, 2023 · 2 comments
Open

Add support for cache eviction policies that can maintain a maximum CI cache size, such as LRU #18294

chrisjrn opened this issue Feb 17, 2023 · 2 comments
Labels
enhancement

Comments

@chrisjrn
Copy link
Contributor

Survey submitter said:

A LRU cache pruning tool. The current CI setup recommends dumping the cache when it grows too large, which causes random spikes as dependencies need to get pulled in again on the next target. It would be great if this ran automatically, such as a "max size" config for the cache.
@huonw
Copy link
Contributor

huonw commented Feb 19, 2023

This may relate to #11167, which seems to have some background thoughts recorded.

cburroughs added a commit to cburroughs/pants that referenced this issue Sep 17, 2024
@cburroughs
Update to Pex 2.19
633ba70 
Highlighted Changelogs:
 * https://github.com/pex-tool/pex/releases/tag/v2.17.0
 * https://github.com/pex-tool/pex/releases/tag/v2.18.0
 * https://github.com/pex-tool/pex/releases/tag/v2.19.0

```
Lockfile diff: 3rdparty/python/user_reqs.lock [python-default]
==                    Upgraded dependencies                     ==

  idna                           3.8          -->   3.10
  pex                            2.16.2       -->   2.19.0
```

ref pantsbuild#18294 pantsbuild#11167
@cburroughs cburroughs mentioned this issue Sep 17, 2024
Merged
cburroughs added a commit to cburroughs/pants that referenced this issue Sep 18, 2024
@cburroughs
Update to Pex 2.19.1
a83a3c0 
Highlighted Changelogs:
 * https://github.com/pex-tool/pex/releases/tag/v2.17.0
 * https://github.com/pex-tool/pex/releases/tag/v2.18.0
 * https://github.com/pex-tool/pex/releases/tag/v2.19.0
 * https://github.com/pex-tool/pex/releases/tag/v2.19.1

```
Lockfile diff: 3rdparty/python/user_reqs.lock [python-default]
==                    Upgraded dependencies                     ==

  idna                           3.8          -->   3.10
  pex                            2.16.2       -->   2.19.1
```

ref pantsbuild#18294 pantsbuild#11167
cburroughs added a commit that referenced this issue Sep 19, 2024
@cburroughs
Update to Pex 2.19.1 (#21425)
Loading
Loading status checks…
ead7b0c 
Highlighted Changelogs:
 * https://github.com/pex-tool/pex/releases/tag/v2.17.0
 * https://github.com/pex-tool/pex/releases/tag/v2.18.0
 * https://github.com/pex-tool/pex/releases/tag/v2.19.0
 * https://github.com/pex-tool/pex/releases/tag/v2.19.1

```
Lockfile diff: 3rdparty/python/user_reqs.lock [python-default]
==                    Upgraded dependencies                     ==

  idna                           3.8          -->   3.10
  pex                            2.16.2       -->   2.19.1
```

ref #18294 #11167
cburroughs added a commit to cburroughs/pants that referenced this issue Dec 6, 2024
@cburroughs
Update to Pex 24.3.1
b21dbad 
Highlighted Changelogs:
 * https://github.com/pex-tool/pex/releases/tag/v2.20.4
 * https://github.com/pex-tool/pex/releases/tag/v2.21.0
 * https://github.com/pex-tool/pex/releases/tag/v2.22.0
 * https://github.com/pex-tool/pex/releases/tag/v2.23.0
 * https://github.com/pex-tool/pex/releases/tag/v2.24.0
 * https://github.com/pex-tool/pex/releases/tag/v2.24.1
 * https://github.com/pex-tool/pex/releases/tag/v2.24.2
 * https://github.com/pex-tool/pex/releases/tag/v2.24.3

```
Lockfile diff: 3rdparty/python/user_reqs.lock [python-default]

==                    Upgraded dependencies                     ==

  cryptography                   43.0.3       -->   44.0.0
  deprecated                     1.2.14       -->   1.2.15
  pex                            2.20.3       -->   2.24.3
  pydantic                       1.10.18      -->   1.10.19
  pyjwt                          2.9.0        -->   2.10.1
  six                            1.16.0       -->   1.17.0
  tomli                          2.0.2        -->   2.2.1
  websockets                     13.1         -->   14.1
  wrapt                          1.16.0       -->   1.17.0
```

ref pantsbuild#11167 pantsbuild#18294 for the `pex3 cache` series of commands
@cburroughs cburroughs mentioned this issue Dec 6, 2024
Merged
cburroughs added a commit to cburroughs/pants that referenced this issue Dec 11, 2024
@cburroughs
Update to Pex 24.3.1
1c04b89 
Highlighted Changelogs:
 * https://github.com/pex-tool/pex/releases/tag/v2.20.4
 * https://github.com/pex-tool/pex/releases/tag/v2.21.0
 * https://github.com/pex-tool/pex/releases/tag/v2.22.0
 * https://github.com/pex-tool/pex/releases/tag/v2.23.0
 * https://github.com/pex-tool/pex/releases/tag/v2.24.0
 * https://github.com/pex-tool/pex/releases/tag/v2.24.1
 * https://github.com/pex-tool/pex/releases/tag/v2.24.2
 * https://github.com/pex-tool/pex/releases/tag/v2.24.3

```
Lockfile diff: 3rdparty/python/user_reqs.lock [python-default]

==                    Upgraded dependencies                     ==

  cryptography                   43.0.3       -->   44.0.0
  deprecated                     1.2.14       -->   1.2.15
  pex                            2.20.3       -->   2.24.3
  pydantic                       1.10.18      -->   1.10.19
  pyjwt                          2.9.0        -->   2.10.1
  six                            1.16.0       -->   1.17.0
  tomli                          2.0.2        -->   2.2.1
  websockets                     13.1         -->   14.1
  wrapt                          1.16.0       -->   1.17.0
```

ref pantsbuild#11167 pantsbuild#18294 for the `pex3 cache` series of commands
cburroughs added a commit that referenced this issue Dec 12, 2024
@cburroughs
Update to Pex 2.24.3 (#21730)
Loading
Loading status checks…
de2a583 
Highlighted Changelogs:
 * https://github.com/pex-tool/pex/releases/tag/v2.20.4
 * https://github.com/pex-tool/pex/releases/tag/v2.21.0
 * https://github.com/pex-tool/pex/releases/tag/v2.22.0
 * https://github.com/pex-tool/pex/releases/tag/v2.23.0
 * https://github.com/pex-tool/pex/releases/tag/v2.24.0
 * https://github.com/pex-tool/pex/releases/tag/v2.24.1
 * https://github.com/pex-tool/pex/releases/tag/v2.24.2
 * https://github.com/pex-tool/pex/releases/tag/v2.24.3

```
Lockfile diff: 3rdparty/python/user_reqs.lock [python-default]

==                    Upgraded dependencies                     ==

  cryptography                   43.0.3       -->   44.0.0
  deprecated                     1.2.14       -->   1.2.15
  pex                            2.20.3       -->   2.24.3
  pydantic                       1.10.18      -->   1.10.19
  pyjwt                          2.9.0        -->   2.10.1
  six                            1.16.0       -->   1.17.0
  tomli                          2.0.2        -->   2.2.1
  websockets                     13.1         -->   14.1
  wrapt                          1.16.0       -->   1.17.0
```

ref #11167 #18294 for the `pex3 cache` series of commands
@jsirois
Copy link
Contributor

jsirois commented Jan 30, 2025

For the Pex cache, see: pex-tool/pex#2586 (comment)
Since Pex 2.24.0 you can run pex3 cache prune to safely prune Pex caches even in the face of parallel Pex runs on the same machine (older than 2 weeks by default): https://github.com/pex-tool/pex/releases/tag/v2.24.0

Pants would have to do very little to expose this since PEX_ROOT=~/.cache/pants/named_caches/pex_root pex3 cache prune does the trick for most Pants setups without using Pants fwict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Projects
2023 Priorities
Status: No status
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jsirois @chrisjrn @huonw