Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fetch: check a cache directory for files matching expected SHAs #143

Merged
merged 5 commits into from
Oct 31, 2022

Conversation

imjasonh
Copy link
Member

@imjasonh imjasonh commented Oct 21, 2022

This adds a --cache-dir flag (default "/var/cache/melange"), which is bound into the build environment at /var/cache/melange (constant), and updates the fetch.yaml pipeline to look for files matching the expected-sha{256,512} in that directory.

The intention is that builds could pre-fetch external dependencies into
the cache directory, then execute builds without hitting the network.

If --cache-dir doesn't exist, it's not bound and fetch will always wget.

Also adds CI checks to build all examples.

Fixes #144

@imjasonh
Copy link
Member Author

This passes tests, but it doesn't actually work. /var/cache/melange is never actually populated inside the build environment, so we never get a cache hit. I'm not sure what I'm missing, I'll take another look in the morning.

@imjasonh imjasonh force-pushed the fetch-cache branch 2 times, most recently from 8f135a4 to e2dfd20 Compare October 21, 2022 14:35
@imjasonh
Copy link
Member Author

This should work now. I can try to add an e2e test that populates and exercises the cache, but it may be tough to prove it doesn't hit the network in GitHub Actions.

This adds a --cache-dir flag (default /var/cache/melange), which is
bound into the build environment at /var/cache/melange (constant), and
updates the fetch.yaml pipeline to look for files matching the
expected-sha{256,512} in that directory.

The intention is that builds could pre-fetch external dependencies into
the cache directory, then execute builds without hitting the network.

If --cache-dir doesn't exist, it's not bound and fetch will always wget.

Also adds CI checks to build all examples.

Signed-off-by: Jason Hall <[email protected]>
Signed-off-by: Jason Hall <[email protected]>
@imjasonh imjasonh mentioned this pull request Oct 21, 2022
@kaniini
Copy link
Contributor

kaniini commented Oct 21, 2022

This should work now. I can try to add an e2e test that populates and exercises the cache, but it may be tough to prove it doesn't hit the network in GitHub Actions.

We can ask bubblewrap to unshare the network namespace.

@imjasonh
Copy link
Member Author

This should work now. I can try to add an e2e test that populates and exercises the cache, but it may be tough to prove it doesn't hit the network in GitHub Actions.

We can ask bubblewrap to unshare the network namespace.

Ooh! A new flag for melange that passes --unshare-net to bwrap?

@kaniini kaniini merged commit dbc0576 into chainguard-dev:main Oct 31, 2022
@imjasonh imjasonh mentioned this pull request Nov 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

sshfs example is broken
2 participants