diff --git a/README.md b/README.md index 7075d45..93bf985 100644 --- a/README.md +++ b/README.md @@ -118,7 +118,7 @@ tags: - 'latest' ``` -This would sync all tags describing versions equal to or larger than `1.31.0`, but lower than `1.31.9`, via the `semver:` filter. The `regex:` filter additionally syncs any `1.26.`*x* image with suffix `-glibc`, `-uclibc`, or `-musl`. Finally, the verbatim tags `1.29.4` and `latest` are also synced. +This syncs all tags describing versions equal to or larger than `1.31.0`, but lower than `1.31.9`, via the `semver:` filter. The `regex:` filter additionally syncs any `1.26.`*x* image with suffix `-glibc`, `-uclibc`, or `-musl`. Finally, the verbatim tags `1.29.4` and `latest` are also synced. Note that the tags of an image need to conform to the *semver* specification *2.0.0* in order to be considered during filtering. The implementation uses the [blang/semver](https://github.com/blang/semver) lib. Have a look at their page or [the GoDoc](https://pkg.go.dev/github.com/blang/semver/v4) for more info on how to write *semver* filter expressions. Semver filtering handles tags starting with a `v` prefix. It also tolerates suffixes, for example platform IDs which are often used in tags, as long as the tag starts with a full *major.minor.patch* semver. Semver **filter expressions** however must not use a `v` prefix or any suffix. @@ -127,15 +127,18 @@ Regex filters use standard *Go* regular expressions. When the first non-whitespa You can add multiple `semver:` and `regex:` filters under `tags`. Note however that the filters are simply ORed, i.e. a tag is synced if it satisfies at least one of the items under `tags`, be it semver, regex, or verbatim. So this is not a filter chain. Also, no sanity checks are done on the filters, so care must be taken to avoid competing or contradicting filters that select all or nothing at all. #### Tag Set Pruning *β feature* -Additionally it is possible to *prune* the resulting tag set with one or more `keep:` filters. These are regular expressions, identical to `regex:` filters (including inversion), but they get applied last, independent of where in the list they appear. If a tag in the filtered tag set does not match **all** of the `keep:` filters, it is removed from the set. This helps in defining tag filters that would be hard to describe with only *semver* and regular expressions. Here's an example for a source registry that attaches OS suffixes to their version tags, such as `2.1.4-buster`: +Additionally it is possible to *prune* the resulting tag set with one or more `keep:` filters. These are regular expressions, identical to `regex:` filters (including inversion), but they get applied last, independent of where in the list they appear. If a tag in the filtered tag set does not match **all** of the `keep:` filters, it is removed from the set. This helps in defining tag filters that would be hard to describe with only *semver* and regular expressions. Note however that `keep:` filters do not apply to verbatim tags! + +Here's an example for a source registry that attaches OS suffixes to their version tags, such as `2.1.4-buster`: ```yaml tags: + - 'latest' - 'semver: >=2.0.0' - 'keep: .+-(alpine|buster)' ``` -This selects all releases starting with version `2.0.0`, but only for the `-alpine` and `-buster` suffixes. +This selects all releases starting with version `2.0.0`, but only for the `-alpine` and `-buster` suffixes. The `latest` tag however is still included in the sync. **Limiting the Tag Count** *α feature* @@ -160,6 +163,30 @@ Keep the following in mind when using tag count limits: - If several `keep: latest` directives are specified in a `tags` list, the last one is used. + +### Tags With Digests *α feature* + +Verbatim tags in a `tags` list may also contain image digests to uniquely identify the requested image. The format for verbatim tags with digests is `[tag@]sha256:{digest value}`, i.e. the tag name can be dropped. As all verbatim tags, they can be mixed with tag filter expressions (see above). If a digest is present, the behavior is as follows: + +- When pulling from the source, the tag is dropped if present, and only the digest used. This is done to achieve consistent behavior between the *Skopeo* and *Docker* relays. + + Background: *Skopeo* currently does not support both tag and digest in the same image reference and exits with an error. For *Docker*, the behavior depends on the version: up through version 1.13.1 and starting again with v20.10.20, *Docker* checks whether tag and digest match and throws an error if they don't. For versions in between, the tag is ignored. + +- When pushing to the target, the name if present is used and the digest dropped. Otherwise the digest is used. + + If there is only a digest, the *Docker* relay auto-generates a tag of the format `dregsy-{digest value}`, since *Docker* does not support pushing by digest-only references. If tags of this format are not desired, specify tags for all digests in your sync config, which would then be used instead. + +Here's an example: + +```yaml +tags: + - 'sha256:1d8a...' + - '1.36.0-uclibc@sha256:58f1...' +``` + +This syncs two distinct versions of an image, according to the given *SHA256* sums. For the second digest in the list, tag `1.36.0-uclibc` is created in the target repository. + + ### Platform Selection (*Multi-Platform* Source Images) *β feature* When the source image is a *multi-platform* image, the platform image adequate for the system on which *dregsy* runs is synced by default. Where this is not applicable, the desired platform can be specified via the `platform` setting, separately for each mapping. To sync all available platform images, `platform: all` can be used. Note however that this shorthand is only supported by the *Skopeo* relay. @@ -236,7 +263,7 @@ If these mechanisms are not applicable in your use case, you can also authentica ```JSON { "username": "oauth2accesstoken", - "password": + "password": "" } ``` diff --git a/cmd/dregsy/main_test.go b/cmd/dregsy/main_test.go index 00d3cb0..83a9bb4 100644 --- a/cmd/dregsy/main_test.go +++ b/cmd/dregsy/main_test.go @@ -207,6 +207,30 @@ func TestE2EDockerTagSetsLimit(t *testing.T) { test.GetParams()) } +// +func TestE2EDockerTagSetsDigest(t *testing.T) { + + // NOTE: Docker does not allow push by digest reference, we therefore + // auto-generate a tag of the form `dregsy-{digest hex}` + + th := test.NewTestHelper(t) + conf := tryConfig(th, "e2e/tagsets/docker-digest.yaml", + 0, 0, true, map[string][]string{ + "tagsets-docker/digest/busybox": { + "dregsy-1d8a02c7a89283870e8dd6bb93dc66bc258e294491a6bbeb193a044ed88773ea", + "1.35.0-uclibc", + }, + }, + test.GetParams()) + + validateDigests(th, conf, map[string][]string{ + "/tagsets-docker/digest/busybox": { + "sha256:1d8a02c7a89283870e8dd6bb93dc66bc258e294491a6bbeb193a044ed88773ea", + "sha256:ff4a7f382ff23a8f716741b6e60ef70a4986af3aff22d26e1f0e0cb4fde29289", + }, + }) +} + // func TestE2ESkopeo(t *testing.T) { tryConfig(test.NewTestHelper(t), "e2e/base/skopeo.yaml", @@ -370,9 +394,29 @@ func TestE2ESkopeoTagSetsLimit(t *testing.T) { test.GetParams()) } +// +func TestE2ESkopeoTagSetsDigest(t *testing.T) { + + th := test.NewTestHelper(t) + conf := tryConfig(th, "e2e/tagsets/skopeo-digest.yaml", + 0, 0, true, map[string][]string{ + "tagsets-skopeo/digest/busybox": { + "1.35.0-uclibc", + }, + }, + test.GetParams()) + + validateDigests(th, conf, map[string][]string{ + "/tagsets-skopeo/digest/busybox": { + "sha256:1d8a02c7a89283870e8dd6bb93dc66bc258e294491a6bbeb193a044ed88773ea", + "sha256:ff4a7f382ff23a8f716741b6e60ef70a4986af3aff22d26e1f0e0cb4fde29289", + }, + }) +} + // func tryConfig(th *test.TestHelper, file string, ticks int, wait time.Duration, - verify bool, expectations map[string][]string, data interface{}) { + verify bool, expectations map[string][]string, data interface{}) *sync.SyncConfig { test.StackTraceDepth = 2 defer func() { test.StackTraceDepth = 1 }() @@ -389,7 +433,7 @@ func tryConfig(th *test.TestHelper, file string, ticks int, wait time.Duration, th.AssertEqual(0, runDregsy(th, ticks, wait, "-config="+dst)) if !verify { - return + return nil } log.Info("TEST - validating result") @@ -401,6 +445,8 @@ func tryConfig(th *test.TestHelper, file string, ticks int, wait time.Duration, } else { validateAgainstTaskMapping(th, c) } + + return c } // @@ -488,6 +534,28 @@ func validatePlatforms(th *test.TestHelper, ref string, task *sync.Task, } } +// +func validateDigests(th *test.TestHelper, c *sync.SyncConfig, + expectations map[string][]string) { + + for _, t := range c.Tasks { + th.AssertNoError(t.Target.RefreshAuth()) + + for _, m := range t.Mappings { + ref := fmt.Sprintf("%s%s", t.Target.Registry, m.To) + + for _, d := range expectations[m.To] { + info, err := skopeo.Inspect( + fmt.Sprintf("%s@%s", ref, d), "", "{{.Digest}}", + util.DecodeJSONAuth(t.Target.GetAuth()), "", + t.Target.SkipTLSVerify) + th.AssertNoError(err) + th.AssertEqual(d, info) + } + } + } +} + // func prepareConfig(src, dst string, data interface{}) error { diff --git a/internal/pkg/relays/docker/docker.go b/internal/pkg/relays/docker/docker.go index 18d2569..8e3da57 100644 --- a/internal/pkg/relays/docker/docker.go +++ b/internal/pkg/relays/docker/docker.go @@ -28,6 +28,7 @@ import ( "github.com/docker/docker/api/types" "github.com/docker/docker/client" "github.com/docker/docker/pkg/jsonmessage" + log "github.com/sirupsen/logrus" "golang.org/x/crypto/ssh/terminal" "github.com/xelalexv/dregsy/internal/pkg/util" @@ -35,20 +36,20 @@ import ( // type image struct { - ID string - Repo string - Path string - Tags []string + id string + reg string + repo string + tags []string } // func (s *image) ref() string { - return fmt.Sprintf("%s/%s", s.Repo, s.Path) + return fmt.Sprintf("%s/%s", s.reg, s.repo) } // func (s *image) refWithTags() string { - return fmt.Sprintf("%s/%s:%v", s.Repo, s.Path, s.Tags) + return fmt.Sprintf("%s/%s:%v", s.reg, s.repo, s.tags) } // @@ -85,8 +86,7 @@ func newEnvClient() (*dockerClient, error) { } // -func (dc *dockerClient) open() error { - var err error +func (dc *dockerClient) open() (err error) { if dc.client == nil { if dc.env { dc.client, err = client.NewEnvClient() @@ -94,92 +94,116 @@ func (dc *dockerClient) open() error { dc.client, err = client.NewClient(dc.host, dc.version, nil, nil) } } - return err + return } // func (dc *dockerClient) ping(attempts int, sleep time.Duration) ( - types.Ping, error) { - var err error + res types.Ping, err error) { + for i := 1; ; i++ { - if res, err := dc.client.Ping(context.Background()); err == nil { - return res, err + if res, err = dc.client.Ping(context.Background()); err == nil { + return } if i >= attempts { break } time.Sleep(sleep) } - return types.Ping{}, - fmt.Errorf( - "unsuccessfully pinged Docker server %d times, last error: %s", - attempts, err) + + return types.Ping{}, fmt.Errorf( + "unsuccessfully pinged Docker server %d times, last error: %s", + attempts, err) } // func (dc *dockerClient) close() error { - var err error if dc.client != nil { - err = dc.client.Close() + return dc.client.Close() } - return err + return nil } // -func (dc *dockerClient) listImages(ref string) ([]*image, error) { +func (dc *dockerClient) listImages(ref string) (list []*image, err error) { + log.WithField("ref", ref).Debug("listing images") imgs, err := dc.client.ImageList( context.Background(), types.ImageListOptions{}) - ret := []*image{} - - if err == nil { - fRepo, fPath, fTag := util.SplitRef(ref) - for _, img := range imgs { - var i *image - for _, rt := range img.RepoTags { - matched, err := match(fRepo, fPath, fTag, rt) - if err != nil { - return ret, err - } - if matched { - repo, path, tag := util.SplitRef(rt) - if i == nil { - i = &image{ - ID: img.ID, - Repo: repo, - Path: path, - } - ret = append(ret, i) + if err != nil { + return + } + + fReg, fRepo, tag := util.SplitRef(ref) // the filter components + name, dig := util.SplitTag(tag) + fTag := name + if dig != "" { + fTag = dig + } + + for _, img := range imgs { + + col := img.RepoTags // switch between the two lists depending on + if dig != "" { // whether we have a digest as tag filter + col = img.RepoDigests + } + + var i *image + + for _, rt := range col { + + if matched, err := match(fReg, fRepo, fTag, rt); err != nil { + return list, err + + } else if matched { + rg, rp, tg := util.SplitRef(rt) + if i == nil { + i = &image{ + id: img.ID, + reg: rg, + repo: rp, } - if tag != "" { - i.Tags = append(i.Tags, tag) + list = append(list, i) + } + if tg != "" { // match has tag + if dig != "" { + // if we're using a digest as filter tag, we need to use + // the full tag from ref, since it may also contain a + // name, which would however be missing in tg; otherwise + // we take the tag as is from the match + tg = tag } + i.tags = append(i.tags, tg) } } } } - return ret, err + return } // -func match(filterRepo, filterPath, filterTag, ref string) (bool, error) { +func match(filterReg, filterRepo, filterTag, ref string) (bool, error) { + + if ref == ":" || ref == "@" { + return false, nil + } - filter := fmt.Sprintf("%s/%s", filterRepo, filterPath) + filter := fmt.Sprintf("%s/%s", filterReg, filterRepo) filterCanon, err := reference.ParseAnyReference(filter) if err != nil { return false, fmt.Errorf("malformed ref in filter '%s', %v", filter, err) } - filterRepo, filterPath, _ = util.SplitRef(filterCanon.String()) + filterReg, filterRepo, _ = util.SplitRef(filterCanon.String()) refCanon, err := reference.ParseAnyReference(ref) if err != nil { return false, fmt.Errorf("malformed image ref '%s': %v", ref, err) } - repo, path, tag := util.SplitRef(refCanon.String()) - return (filterRepo == "" || filterRepo == repo) && - (filterPath == "" || filterPath == path) && + reg, repo, tag := util.SplitRef(refCanon.String()) + return (filterReg == "" || filterReg == reg) && + (filterRepo == "" || filterRepo == repo) && (filterTag == "" || filterTag == tag), nil } @@ -216,8 +240,7 @@ func (dc *dockerClient) tagImage(source, target string) error { } // -func (dc *dockerClient) handleLog(rc io.ReadCloser, err error, - verbose bool) error { +func (dc *dockerClient) handleLog(rc io.ReadCloser, err error, verbose bool) error { if err != nil { return err diff --git a/internal/pkg/relays/docker/dockerrelay.go b/internal/pkg/relays/docker/dockerrelay.go index 2b1d920..cf2c74b 100644 --- a/internal/pkg/relays/docker/dockerrelay.go +++ b/internal/pkg/relays/docker/dockerrelay.go @@ -116,17 +116,18 @@ func (r *DockerRelay) Sync(opt *relays.SyncOptions) error { var err error // When no tags are specified, a simple docker pull without a tag will get - // all tags. So for Docker relay, we don't need to list tags in this case. + // all tags. So for that case, we don't need to list tags. + if !opt.Tags.IsEmpty() { - srcCertDir := "" + var certs string repo, _, _ := util.SplitRef(opt.SrcRef) if repo != "" { - srcCertDir = skopeo.CertsDirForRepo(repo) + certs = skopeo.CertsDirForRepo(repo) } tags, err = opt.Tags.Expand(func() ([]string, error) { return skopeo.ListAllTags( opt.SrcRef, util.DecodeJSONAuth(opt.SrcAuth), - srcCertDir, opt.SrcSkipTLSVerify) + certs, opt.SrcSkipTLSVerify) }) if err != nil { @@ -134,43 +135,51 @@ func (r *DockerRelay) Sync(opt *relays.SyncOptions) error { } } - if len(tags) == 0 { + if len(tags) == 0 { // pull all tags if err = r.pull(opt.SrcRef, opt.Platform, opt.SrcAuth, true, opt.Verbose); err != nil { return fmt.Errorf( "error pulling source image '%s': %v", opt.SrcRef, err) } - } else { + } else { // pull tag by tag; tags with digest are pulled by digest only for _, tag := range tags { - srcRefTagged := fmt.Sprintf("%s:%s", opt.SrcRef, tag) - if err = r.pull(srcRefTagged, opt.Platform, opt.SrcAuth, + tagged, _ := util.JoinRefsAndTag(opt.SrcRef, "", tag) + if err = r.pull(tagged, opt.Platform, opt.SrcAuth, false, opt.Verbose); err != nil { return fmt.Errorf( - "error pulling source image '%s': %v", srcRefTagged, err) + "error pulling source image '%s': %v", tagged, err) } } } + // Now that we've pulled all required tags, we need to get the image IDs + // for pushing to the target registry. For this, we list all images that are + // currently present, and then filter by registry, repo & tags. We do this + // rather than using filter expressions in Docker list options, to maintain + // better control over the reference matching. Docker may refer to images + // originating from DockerHub in the short form, which would not match fully + // qualified references that could be used in the sync config. In our own + // filtering, we canonicalize all references before matching. + log.Info("relevant tags:") var srcImages []*image - if len(tags) == 0 { + if len(tags) == 0 { // use all local images that match source reference srcImages, err = r.list(opt.SrcRef) if err != nil { log.Errorf("error listing all tags of source image '%s': %v", opt.SrcRef, err) } - } else { + } else { // filter local images by source reference and tags for _, tag := range tags { - srcRefTagged := fmt.Sprintf("%s:%s", opt.SrcRef, tag) - srcImageTagged, err := r.list(srcRefTagged) + tagged := util.JoinRefAndTag(opt.SrcRef, tag) + imgs, err := r.list(tagged) if err != nil { - log.Errorf( - "error listing source image '%s': %v", srcRefTagged, err) + log.Errorf("error listing source image '%s': %v", tagged, err) } - srcImages = append(srcImages, srcImageTagged...) + srcImages = append(srcImages, imgs...) } } @@ -180,8 +189,8 @@ func (r *DockerRelay) Sync(opt *relays.SyncOptions) error { log.WithField("ref", opt.TrgtRef).Info("setting tags for target image") - _, err = r.tag(srcImages, opt.TrgtRef) - if err != nil { + // We now tag the source images for the target registry. + if err = r.tag(srcImages, opt.TrgtRef); err != nil { return fmt.Errorf("error setting tags: %v", err) } @@ -189,6 +198,12 @@ func (r *DockerRelay) Sync(opt *relays.SyncOptions) error { "ref": opt.TrgtRef, "platform": opt.Platform}).Info("pushing target image") + // Finally, the images are pushed to the target. This includes all present + // tags. + // + // FIXME: target tags should be removed to not interfere with tag count + // limiting + // if err := r.push( opt.TrgtRef, opt.Platform, opt.TrgtAuth, opt.Verbose); err != nil { return fmt.Errorf("error pushing target image: %v", err) @@ -208,32 +223,48 @@ func (r *DockerRelay) list(ref string) ([]*image, error) { } // -func (r *DockerRelay) tag(images []*image, targetRef string) ( - []*image, error) { - - taggedImages := []*image{} - targetRepo, targetPath, _ := util.SplitRef(targetRef) +func (r *DockerRelay) tag(images []*image, targetRef string) error { for _, img := range images { - tagged := &image{ - ID: img.ID, - Repo: targetRepo, - Path: targetPath, - Tags: img.Tags, - } - for _, tag := range img.Tags { - if err := r.client.tagImage(img.ID, fmt.Sprintf("%s:%s", - tagged.ref(), tag)); err != nil { - return nil, err + for _, tag := range img.tags { + if tag != "" { + n, d := util.SplitTag(tag) + if n == "" { + // Docker does not support pushing by digest only ref; we + // therefore auto-generate a tag using the digest value + log.Debug("generating tag for digest only ref") + n = tagFromDigest(d) + } + + log.WithFields( + log.Fields{"ref": targetRef, "tag": n}).Debug("tagging") + + if err := r.client.tagImage( + img.id, fmt.Sprintf("%s:%s", targetRef, n)); err != nil { + return err + } } } - taggedImages = append(taggedImages, tagged) } - return taggedImages, nil + return nil } // func (r *DockerRelay) push(ref, platform, auth string, verbose bool) error { return r.client.pushImage(ref, true, platform, auth, verbose) } + +// +func tagFromDigest(d string) string { + + if util.IsDigest(d) { + d = d[len(util.DigestPrefix):] + } + + ret := fmt.Sprintf("dregsy-%s", d) + if len(ret) > 128 { + ret = ret[:128] + } + return ret +} diff --git a/internal/pkg/relays/skopeo/skopeorelay.go b/internal/pkg/relays/skopeo/skopeorelay.go index 3e71540..65c7846 100644 --- a/internal/pkg/relays/skopeo/skopeorelay.go +++ b/internal/pkg/relays/skopeo/skopeorelay.go @@ -139,9 +139,9 @@ func (r *SkopeoRelay) Sync(opt *relays.SyncOptions) error { log.WithFields( log.Fields{"tag": t, "platform": opt.Platform}).Info("syncing tag") + src, trgt := util.JoinRefsAndTag(opt.SrcRef, opt.TrgtRef, t) rc := append(cmd, - fmt.Sprintf("docker://%s:%s", opt.SrcRef, t), - fmt.Sprintf("docker://%s:%s", opt.TrgtRef, t)) + fmt.Sprintf("docker://%s", src), fmt.Sprintf("docker://%s", trgt)) switch opt.Platform { case "": diff --git a/internal/pkg/tags/tagset.go b/internal/pkg/tags/tagset.go index 92688f6..3da7e57 100644 --- a/internal/pkg/tags/tagset.go +++ b/internal/pkg/tags/tagset.go @@ -200,12 +200,7 @@ func (ts *TagSet) Expand(lister func() ([]string, error)) ([]string, error) { } } - if ts.HasVerbatim() { - log.Debugf("verbatim tags: %v", ts.verbatim) - addToSet(set, ts.verbatim) - } - - ret := make([]string, 0, len(set)) + ret := make([]string, 0, len(set)+len(ts.verbatim)) var pruned []string for t := range set { @@ -220,6 +215,12 @@ func (ts *TagSet) Expand(lister func() ([]string, error)) ([]string, error) { log.Debugf("pruned tags: %v", pruned) + // verbatim tags must not be pruned, but are subject to count limiting + if ts.HasVerbatim() { + log.Debugf("adding verbatim tags: %v", ts.verbatim) + ret = append(ret, ts.verbatim...) + } + if ts.keepCount > 0 { log.WithField("limit", ts.keepCount).Debug("reducing tag set") ret = ts.reduce(ret, ts.keepCount) diff --git a/internal/pkg/util/util.go b/internal/pkg/util/util.go index 3ae10e8..2bbdd33 100644 --- a/internal/pkg/util/util.go +++ b/internal/pkg/util/util.go @@ -29,28 +29,104 @@ import ( ) // -func SplitRef(ref string) (repo, path, tag string) { +const DigestPrefix = "sha256:" + +const FormatDigest = "%s@%s" +const FormatName = "%s:%s" + +// +func SplitRef(ref string) (reg, repo, tag string) { ix := strings.Index(ref, "/") if ix == -1 { - repo = "" - path = ref + reg = "" + repo = ref } else { - repo = ref[:ix] - path = ref[ix+1:] + reg = ref[:ix] + repo = ref[ix+1:] } - ix = strings.Index(path, ":") + // note: if ref contains a colon for specifying registry port, it is left of + // the first slash, and hence no longer included in repo at this point + ixC := strings.Index(repo, ":") + ixA := strings.Index(repo, "@") - if ix > -1 { - tag = path[ix+1:] - path = path[:ix] + if ixC > -1 && ixA > -1 { // we have both : and @ + if ixC < ixA { + ix = ixC + } else { + ix = ixA + } + } else if ixA > -1 { // only @ (actually invalid) + ix = ixA + } else if ixC > -1 { // only : + ix = ixC + } else { + return // no tag + } + + tag = repo[ix+1:] + repo = repo[:ix] + + return +} + +// HasName returns true if tag HAS or IS a name +func HasName(tag string) bool { + n, _ := SplitTag(tag) + return n != "" +} + +// HasDigest returns true if tag HAS or IS a digest +func HasDigest(tag string) bool { + _, d := SplitTag(tag) + return d != "" +} + +// IsDigest returns true if d is a digest string. For performance reasons, this +// currently only checks for the presence of the `sha256:` prefix, and does not +// run a regex against d for full compliance check. This way, incorrect digests +// lead to errors, but correct digests do not incur a computational overhead. +func IsDigest(d string) bool { + return strings.HasPrefix(d, DigestPrefix) +} + +// +func SplitTag(tag string) (name, digest string) { + + if strings.HasPrefix(tag, ":") { + tag = tag[1:] + } + if tag == "" { + return + } + + if p := strings.Split(tag, "@"); len(p) == 1 { // either tag or digest + if IsDigest(p[0]) { + digest = p[0] + } else { + name = p[0] + } + } else { // both + name = p[0] + digest = p[1] } return } +// +func JoinTag(name, digest string) string { + if digest == "" { + return name + } + if name == "" { + return digest + } + return fmt.Sprintf(FormatDigest, name, digest) +} + // func SplitPlatform(p string) (os, arch, variant string) { @@ -74,6 +150,41 @@ func SplitPlatform(p string) (os, arch, variant string) { return } +// JoinRefAndTag joins ref with tag, inserting the correct separator ':' or '@', +// depending on whether tag contains a name part or is purely a digest. +func JoinRefAndTag(ref, tag string) string { + if HasName(tag) { + return fmt.Sprintf(FormatName, ref, tag) + } + return fmt.Sprintf(FormatDigest, ref, tag) +} + +// JoinRefsAndTag joins the source and target ref for a sync action each with +// tag, according to these rules: +// +// - If tag contains a digest, srcRef is joined with only the digest as tag, and +// trgtRef with either only the name part of tag (if present), or the digest. +// +// - Otherwise, srcRef and trgtRef are joined with tag. +// +// This ensures that if a digest is present, we always use that when pulling an +// image, but still use the name if present when pushing. +// +func JoinRefsAndTag(srcRef, trgtRef, tag string) (src, trgt string) { + if name, digest := SplitTag(tag); digest != "" { + src = fmt.Sprintf(FormatDigest, srcRef, digest) + if name != "" { + trgt = fmt.Sprintf(FormatName, trgtRef, name) + } else { + trgt = fmt.Sprintf(FormatDigest, trgtRef, digest) + } + } else { + src = fmt.Sprintf(FormatName, srcRef, name) + trgt = fmt.Sprintf(FormatName, trgtRef, name) + } + return +} + // type creds struct { Username string diff --git a/test/fixtures/e2e/tagsets/docker-digest.yaml b/test/fixtures/e2e/tagsets/docker-digest.yaml new file mode 100644 index 0000000..a90ab33 --- /dev/null +++ b/test/fixtures/e2e/tagsets/docker-digest.yaml @@ -0,0 +1,23 @@ +relay: docker + +docker: + dockerhost: {{ .DockerHost }} + +tasks: +- name: test-docker-digest + verbose: true + source: + registry: registry.hub.docker.com + auth: {{ .DockerhubAuth }} + target: + registry: 127.0.0.1:5000 + auth: {{ .LocalAuth }} + # not actually supported for Docker relay, but need this for validation + skip-tls-verify: true + mappings: + - from: library/busybox + to: tagsets-docker/digest/busybox + tags: + - 'sha256:1d8a02c7a89283870e8dd6bb93dc66bc258e294491a6bbeb193a044ed88773ea' # 1.36.0-musl + - '1.35.0-uclibc@sha256:ff4a7f382ff23a8f716741b6e60ef70a4986af3aff22d26e1f0e0cb4fde29289' + - 'keep: all.+' # make sure digests do not get removed by tag pruning diff --git a/test/fixtures/e2e/tagsets/skopeo-digest.yaml b/test/fixtures/e2e/tagsets/skopeo-digest.yaml new file mode 100644 index 0000000..311e60d --- /dev/null +++ b/test/fixtures/e2e/tagsets/skopeo-digest.yaml @@ -0,0 +1,19 @@ +relay: skopeo + +tasks: +- name: test-skopeo-digest + verbose: true + source: + registry: registry.hub.docker.com + auth: {{ .DockerhubAuth }} + target: + registry: 127.0.0.1:5000 + auth: {{ .LocalAuth }} + skip-tls-verify: true + mappings: + - from: library/busybox + to: tagsets-skopeo/digest/busybox + tags: + - 'sha256:1d8a02c7a89283870e8dd6bb93dc66bc258e294491a6bbeb193a044ed88773ea' # 1.36.0-musl + - '1.35.0-uclibc@sha256:ff4a7f382ff23a8f716741b6e60ef70a4986af3aff22d26e1f0e0cb4fde29289' + - 'keep: all.+' # make sure digests do not get removed by tag pruning