support local cache exporter and importer #615

AkihiroSuda · 2018-09-11T10:27:34Z

Local exporter/importer: Support index.json
CLI: Support specifying multiple cache exporters/importers
~~Solver: define build caps~~ (do we need caps?)
*: add tests

Impact on CLI

Old (deprecated but still effective): --export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max
New: --export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max

Impact on API

New fields are added to control.proto and gateway.proto. The daemon
internally translates old API calls to the new ones.
While new API can be used for registry caches, the client continues to use the legacy API for registry caches to ensure compatibility with old daemons.
To import local caches with a frontend, the frontend needs to support a new frontend opt cache-imports.

Usage

To/From registry

buildctl build ... --export-cache type=registry,ref=localhost:5000/myrepo:buildcache
buildctl build ... --import-cache type=registry,ref=localhost:5000/myrepo:buildcache

To/From local filesystem

buildctl build ... --export-cache type=local,store=path/to/output-dir
buildctl build ... --import-cache type=local,store=path/to/input-dir

cache/remotecache/local/local.go

tonistiigi · 2018-09-17T17:28:43Z

With the current solution if we don't use specific refs we need to pull all the cache metadata every time (a big json structure). I'm not sure if this could cause scaling issues here yet but if we would switch the backend from files to some highly available cache service that is exposed to multiple users we would probably need a solution for that. I'm ok with making cache.Query async for that in solver but we should figure out some protocol for how these requests are made to remote parties. Again, not that important for the file-backed sources unless we want to put this data in a boltdb/sqlite on the client side(and we probably don't want to do that).

tonistiigi · 2018-10-12T22:47:51Z

@AkihiroSuda What are the next steps for this?

AkihiroSuda · 2018-10-15T05:19:37Z

Convention for the name of lock file for index.json. Maybe just index.json.lock.
Syntax for specifying "type" ("registry" and "local". In future, we can add "containerd", "another-buildkitd-instance",...). My suggestion is to use "CSV" style.
Syntax for exposting without tag (even without "latest"). My suggestion is to use CSV styel for this syntax as well. e.g.
--export-cache ref=localhost:5000/foo/bar,type=registry,no-index

AkihiroSuda · 2018-10-23T00:39:54Z

Any thought about CSV?

tonistiigi · 2018-10-24T05:55:48Z

Not strictly against csv but seems incompatible with the other flags of buildctl eg. frontend-, exporter-. Or are you suggesting to change these flags as well?

AkihiroSuda · 2018-10-26T07:45:19Z

Can we deprecate --export-cache and introduce --cache-exporter for consistency with --exporter and --frontend?

e.g.
--cache-exporter registry --cache-exporter-opt name=localhost:5000/foo/bar (for consistency with --exporter, maybe s/registry/image/ ? not sure..)

--cache-exporter local --cache-exporter-opt output=path/to/output-dir

tonistiigi · 2018-10-26T15:41:25Z

Can we deprecate --export-cache and introduce --cache-exporter for consistency with --exporter and --frontend?

Yes, shouldn't be a problem.

AkihiroSuda · 2018-11-21T03:33:21Z

Sorry for delay, I'll try to work on next month

AkihiroSuda · 2019-01-02T22:37:48Z

Implementation is WIP but CLI and API are ready to review.

README.md

AkihiroSuda · 2019-01-06T13:14:01Z

updated PR with support for index.json

AkihiroSuda · 2019-01-07T02:14:20Z

Updated: now session/content can aggregate map[string]content.Store with gRPC metadata string. i.e. --import-cache and --export-cache can refer to different directories

api/services/control/control.proto

client/graph.go

client/ociindex/ociindex.go

client/solve.go

frontend/gateway/pb/gateway.proto

tonistiigi · 2019-01-09T02:44:55Z

client/client_test.go

@@ -1156,14 +1157,8 @@ func testBuildPushAndValidate(t *testing.T, sb integration.Sandbox) {
 	require.False(t, ok)
 }

-func testBasicCacheImportExport(t *testing.T, sb integration.Sandbox) {
+func testBasicCacheImportExport(t *testing.T, sb integration.Sandbox, cacheOptionsEntry CacheOptionsEntry) {


possible follow-up: we need to update this test(or add a new one) to include local sources. Currently not testing a lot of the exporter/cachekeys code.

tonistiigi · 2019-01-09T02:48:01Z

README.md

+* `mode=min` (default): only export layers for the resulting image
+* `mode=max`: export all the layers of all intermediate steps
+* `ref=docker.io/user/image:tag`: reference for `registry` cache exporter
+* `store=path/to/output-dir`: directory for `local` cache exporter


why isn't tag (or name) a supported import/export option for local?

can be follow-up?

README.md

AkihiroSuda · 2019-01-15T11:51:59Z

updated

frontend/gateway/grpcclient/client.go

Export: $ buildctl build ... --export-cache type=local,store=/path/to/output-dir Import: $ buildctl build ... --import-cache type=local,store=/path/to/input-dir Impact on CLI: * Old (deprecated but still effective): `--export-cache localhost:5000/myrepo:buildcache --export-cache-opt mode=max` * New: `--export-cache type=registry,ref=localhost:5000/myrepo:buildcache,mode=max` Impact on API: * New fields are added to control.proto and gateway.proto. The daemon internally translates old API calls to the new ones. * While new API can be used for `registry` caches, the client continues to use the legacy API for `registry` caches to ensure compatibility with old daemons. * To import `local` caches with a frontend, the frontend needs to support a new frontend opt `cache-imports`. Signed-off-by: Akihiro Suda <[email protected]>

AkihiroSuda · 2019-01-18T17:38:37Z

updated

AkihiroSuda added the status/do-not-merge label Sep 11, 2018

ijc reviewed Sep 11, 2018

View reviewed changes

cache/remotecache/local/local.go Outdated Show resolved Hide resolved

tonistiigi mentioned this pull request Oct 24, 2018

How to export cache in buildkit #699

Closed

AkihiroSuda force-pushed the export-cache-local branch 7 times, most recently from d84ea44 to 793fe1a Compare January 2, 2019 19:53

AkihiroSuda added this to the v0.4.0 milestone Jan 2, 2019

AkihiroSuda force-pushed the export-cache-local branch 5 times, most recently from e3d533f to f59455f Compare January 2, 2019 22:31

AkihiroSuda force-pushed the export-cache-local branch 2 times, most recently from 77c8143 to f1615d8 Compare January 2, 2019 23:54

tonistiigi reviewed Jan 3, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

AkihiroSuda force-pushed the export-cache-local branch 2 times, most recently from 1d1684b to 1b87099 Compare January 6, 2019 06:56

AkihiroSuda force-pushed the export-cache-local branch 3 times, most recently from 51912eb to a49a1a5 Compare January 6, 2019 13:11

AkihiroSuda changed the title ~~[WIP/DNM/RFC] support local cache exporter and importer~~ support local cache exporter and importer Jan 6, 2019

AkihiroSuda removed the status/do-not-merge label Jan 6, 2019

AkihiroSuda force-pushed the export-cache-local branch 2 times, most recently from c45cf8d to 1fea4ae Compare January 7, 2019 02:12

AkihiroSuda force-pushed the export-cache-local branch from 1fea4ae to 111a38f Compare January 7, 2019 05:47

tonistiigi mentioned this pull request Jan 9, 2019

allow exporting cache metadata in the image config #777

Merged

tonistiigi reviewed Jan 9, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

tonistiigi mentioned this pull request Jan 10, 2019

Make exported build cache deterministic #780

Merged

AkihiroSuda force-pushed the export-cache-local branch 3 times, most recently from 32a3b1a to aad7b22 Compare January 15, 2019 11:51

AkihiroSuda force-pushed the export-cache-local branch from aad7b22 to 6bbf0cd Compare January 15, 2019 11:52

tonistiigi reviewed Jan 15, 2019

View reviewed changes

frontend/gateway/grpcclient/client.go Outdated Show resolved Hide resolved

AkihiroSuda force-pushed the export-cache-local branch from 6bbf0cd to 6e7617e Compare January 18, 2019 17:38

tonistiigi approved these changes Jan 18, 2019

View reviewed changes

tonistiigi merged commit 3cba037 into moby:master Jan 18, 2019

AkihiroSuda mentioned this pull request Feb 18, 2019

buildctl: new CLI ("Option C+") #807

Merged

mortargrind mentioned this pull request Apr 20, 2022

Supporting multiple cache exports #2818

Closed

jedevc mentioned this pull request May 24, 2023

llbsolver: move history blobs to a separate namespace #3833

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

support local cache exporter and importer #615

support local cache exporter and importer #615

AkihiroSuda commented Sep 11, 2018 •

edited

Loading

tonistiigi commented Sep 17, 2018

tonistiigi commented Oct 12, 2018

AkihiroSuda commented Oct 15, 2018

AkihiroSuda commented Oct 23, 2018

tonistiigi commented Oct 24, 2018

AkihiroSuda commented Oct 26, 2018

tonistiigi commented Oct 26, 2018

AkihiroSuda commented Nov 21, 2018

AkihiroSuda commented Jan 2, 2019

AkihiroSuda commented Jan 6, 2019

AkihiroSuda commented Jan 7, 2019 •

edited

Loading

tonistiigi Jan 9, 2019

tonistiigi Jan 9, 2019

AkihiroSuda Jan 15, 2019

AkihiroSuda commented Jan 15, 2019

AkihiroSuda commented Jan 18, 2019

support local cache exporter and importer #615

support local cache exporter and importer #615

Conversation

AkihiroSuda commented Sep 11, 2018 • edited Loading

Impact on CLI

Impact on API

Usage

To/From registry

To/From local filesystem

tonistiigi commented Sep 17, 2018

tonistiigi commented Oct 12, 2018

AkihiroSuda commented Oct 15, 2018

AkihiroSuda commented Oct 23, 2018

tonistiigi commented Oct 24, 2018

AkihiroSuda commented Oct 26, 2018

tonistiigi commented Oct 26, 2018

AkihiroSuda commented Nov 21, 2018

AkihiroSuda commented Jan 2, 2019

AkihiroSuda commented Jan 6, 2019

AkihiroSuda commented Jan 7, 2019 • edited Loading

tonistiigi Jan 9, 2019

Choose a reason for hiding this comment

tonistiigi Jan 9, 2019

Choose a reason for hiding this comment

AkihiroSuda Jan 15, 2019

Choose a reason for hiding this comment

AkihiroSuda commented Jan 15, 2019

AkihiroSuda commented Jan 18, 2019

AkihiroSuda commented Sep 11, 2018 •

edited

Loading

AkihiroSuda commented Jan 7, 2019 •

edited

Loading