#86 support digests in tags

README.md

@@ -118,7 +118,7 @@ tags:
   - 'latest'
 ```
 
-This would sync all tags describing versions equal to or larger than `1.31.0`, but lower than `1.31.9`, via the `semver:` filter. The `regex:` filter additionally syncs any `1.26.`*x* image with suffix `-glibc`, `-uclibc`, or `-musl`. Finally, the verbatim tags `1.29.4` and `latest` are also synced.
+This syncs all tags describing versions equal to or larger than `1.31.0`, but lower than `1.31.9`, via the `semver:` filter. The `regex:` filter additionally syncs any `1.26.`*x* image with suffix `-glibc`, `-uclibc`, or `-musl`. Finally, the verbatim tags `1.29.4` and `latest` are also synced.
 
 Note that the tags of an image need to conform to the *semver* specification *2.0.0* in order to be considered during filtering. The implementation uses the [blang/semver](https://github.com/blang/semver) lib. Have a look at their page or [the GoDoc](https://pkg.go.dev/github.com/blang/semver/v4) for more info on how to write *semver* filter expressions. Semver filtering handles tags starting with a `v` prefix. It also tolerates suffixes, for example platform IDs which are often used in tags, as long as the tag starts with a full *major.minor.patch* semver. Semver **filter expressions** however must not use a `v` prefix or any suffix.
 
@@ -127,15 +127,18 @@ Regex filters use standard *Go* regular expressions. When the first non-whitespa
 You can add multiple `semver:` and `regex:` filters under `tags`. Note however that the filters are simply ORed, i.e. a tag is synced if it satisfies at least one of the items under `tags`, be it semver, regex, or verbatim. So this is not a filter chain. Also, no sanity checks are done on the filters, so care must be taken to avoid competing or contradicting filters that select all or nothing at all.
 
 #### Tag Set Pruning <sup>*&#946; feature*</sup>
-Additionally it is possible to *prune* the resulting tag set with one or more `keep:` filters. These are regular expressions, identical to `regex:` filters (including inversion), but they get applied last, independent of where in the list they appear. If a tag in the filtered tag set does not match **all** of the `keep:` filters, it is removed from the set. This helps in defining tag filters that would be hard to describe with only *semver* and regular expressions. Here's an example for a source registry that attaches OS suffixes to their version tags, such as `2.1.4-buster`:
+Additionally it is possible to *prune* the resulting tag set with one or more `keep:` filters. These are regular expressions, identical to `regex:` filters (including inversion), but they get applied last, independent of where in the list they appear. If a tag in the filtered tag set does not match **all** of the `keep:` filters, it is removed from the set. This helps in defining tag filters that would be hard to describe with only *semver* and regular expressions. Note however that `keep:` filters do not apply to verbatim tags!
+
+Here's an example for a source registry that attaches OS suffixes to their version tags, such as `2.1.4-buster`:
 
 ```yaml
 tags:
+  - 'latest'
   - 'semver: >=2.0.0'
   - 'keep: .+-(alpine|buster)'
 ```
 
-This selects all releases starting with version `2.0.0`, but only for the `-alpine` and `-buster` suffixes.
+This selects all releases starting with version `2.0.0`, but only for the `-alpine` and `-buster` suffixes. The `latest` tag however is still included in the sync.
 
 **Limiting the Tag Count** <sup>*&#945; feature*</sup>
 
@@ -160,6 +163,30 @@ Keep the following in mind when using tag count limits:
 
 - If several `keep: latest` directives are specified in a `tags` list, the last one is used. 
 
+
+### Tags With Digests <sup>*&#945; feature*</sup>
+
+Verbatim tags in a `tags` list may also contain image digests to uniquely identify the requested image. The format for verbatim tags with digests is `[tag@]sha256:{digest value}`, i.e. the tag name can be dropped. As all verbatim tags, they can be mixed with tag filter expressions (see above). If a digest is present, the behavior is as follows:
+
+- When pulling from the source, the tag is dropped if present, and only the digest used. This is done to achieve consistent behavior between the *Skopeo* and *Docker* relays.
+
+    Background: *Skopeo* currently does not support both tag and digest in the same image reference and exits with an error. For *Docker*, the behavior depends on the version: up through version 1.13.1 and starting again with v20.10.20, *Docker* checks whether tag and digest match and throws an error if they don't. For versions in between, the tag is ignored.
+
+- When pushing to the target, the name if present is used and the digest dropped. Otherwise the digest is used.
+
+    If there is only a digest, the *Docker* relay auto-generates a tag of the format `dregsy-{digest value}`, since *Docker* does not support pushing by digest-only references. If tags of this format are not desired, specify tags for all digests in your sync config, which would then be used instead.
+
+Here's an example:
+
+```yaml
+tags:
+  - 'sha256:1d8a...'
+  - '1.36.0-uclibc@sha256:58f1...'
+```
+
+This syncs two distinct versions of an image, according to the given *SHA256* sums. For the second digest in the list, tag `1.36.0-uclibc` is created in the target repository.
+
+
 ### Platform Selection (*Multi-Platform* Source Images) <sup>*&#946; feature*</sup>
 
 When the source image is a *multi-platform* image, the platform image adequate for the system on which *dregsy* runs is synced by default. Where this is not applicable, the desired platform can be specified via the `platform` setting, separately for each mapping. To sync all available platform images, `platform: all` can be used. Note however that this shorthand is only supported by the *Skopeo* relay.
@@ -236,7 +263,7 @@ If these mechanisms are not applicable in your use case, you can also authentica
 ```JSON
 {
     "username": "oauth2accesstoken",
-    "password": <oauth2 token as described in the Artifact Registry documentation>
+    "password": "<oauth2 token as described in the Artifact Registry documentation>"
 }
 ```
 

cmd/dregsy/main_test.go

@@ -207,6 +207,30 @@ func TestE2EDockerTagSetsLimit(t *testing.T) {
 		test.GetParams())
 }
 
+//
+func TestE2EDockerTagSetsDigest(t *testing.T) {
+
+	// NOTE: Docker does not allow push by digest reference, we therefore
+	//       auto-generate a tag of the form `dregsy-{digest hex}`
+
+	th := test.NewTestHelper(t)
+	conf := tryConfig(th, "e2e/tagsets/docker-digest.yaml",
+		0, 0, true, map[string][]string{
+			"tagsets-docker/digest/busybox": {
+				"dregsy-1d8a02c7a89283870e8dd6bb93dc66bc258e294491a6bbeb193a044ed88773ea",
+				"1.35.0-uclibc",
+			},
+		},
+		test.GetParams())
+
+	validateDigests(th, conf, map[string][]string{
+		"/tagsets-docker/digest/busybox": {
+			"sha256:1d8a02c7a89283870e8dd6bb93dc66bc258e294491a6bbeb193a044ed88773ea",
+			"sha256:ff4a7f382ff23a8f716741b6e60ef70a4986af3aff22d26e1f0e0cb4fde29289",
+		},
+	})
+}
+
 //
 func TestE2ESkopeo(t *testing.T) {
 	tryConfig(test.NewTestHelper(t), "e2e/base/skopeo.yaml",
@@ -370,9 +394,29 @@ func TestE2ESkopeoTagSetsLimit(t *testing.T) {
 		test.GetParams())
 }
 
+//
+func TestE2ESkopeoTagSetsDigest(t *testing.T) {
+
+	th := test.NewTestHelper(t)
+	conf := tryConfig(th, "e2e/tagsets/skopeo-digest.yaml",
+		0, 0, true, map[string][]string{
+			"tagsets-skopeo/digest/busybox": {
+				"1.35.0-uclibc",
+			},
+		},
+		test.GetParams())
+
+	validateDigests(th, conf, map[string][]string{
+		"/tagsets-skopeo/digest/busybox": {
+			"sha256:1d8a02c7a89283870e8dd6bb93dc66bc258e294491a6bbeb193a044ed88773ea",
+			"sha256:ff4a7f382ff23a8f716741b6e60ef70a4986af3aff22d26e1f0e0cb4fde29289",
+		},
+	})
+}
+
 //
 func tryConfig(th *test.TestHelper, file string, ticks int, wait time.Duration,
-	verify bool, expectations map[string][]string, data interface{}) {
+	verify bool, expectations map[string][]string, data interface{}) *sync.SyncConfig {
 
 	test.StackTraceDepth = 2
 	defer func() { test.StackTraceDepth = 1 }()
@@ -389,7 +433,7 @@ func tryConfig(th *test.TestHelper, file string, ticks int, wait time.Duration,
 	th.AssertEqual(0, runDregsy(th, ticks, wait, "-config="+dst))
 
 	if !verify {
-		return
+		return nil
 	}
 
 	log.Info("TEST - validating result")
@@ -401,6 +445,8 @@ func tryConfig(th *test.TestHelper, file string, ticks int, wait time.Duration,
 	} else {
 		validateAgainstTaskMapping(th, c)
 	}
+
+	return c
 }
 
 //
@@ -488,6 +534,28 @@ func validatePlatforms(th *test.TestHelper, ref string, task *sync.Task,
 	}
 }
 
+//
+func validateDigests(th *test.TestHelper, c *sync.SyncConfig,
+	expectations map[string][]string) {
+
+	for _, t := range c.Tasks {
+		th.AssertNoError(t.Target.RefreshAuth())
+
+		for _, m := range t.Mappings {
+			ref := fmt.Sprintf("%s%s", t.Target.Registry, m.To)
+
+			for _, d := range expectations[m.To] {
+				info, err := skopeo.Inspect(
+					fmt.Sprintf("%s@%s", ref, d), "", "{{.Digest}}",
+					util.DecodeJSONAuth(t.Target.GetAuth()), "",
+					t.Target.SkipTLSVerify)
+				th.AssertNoError(err)
+				th.AssertEqual(d, info)
+			}
+		}
+	}
+}
+
 //
 func prepareConfig(src, dst string, data interface{}) error {
 

internal/pkg/relays/docker/docker.go

@@ -28,27 +28,28 @@ import (
 	"github.com/docker/docker/api/types"
 	"github.com/docker/docker/client"
 	"github.com/docker/docker/pkg/jsonmessage"
+	log "github.com/sirupsen/logrus"
 	"golang.org/x/crypto/ssh/terminal"
 
 	"github.com/xelalexv/dregsy/internal/pkg/util"
 )
 
 //
 type image struct {
-	ID   string
-	Repo string
-	Path string
-	Tags []string
+	id   string
+	reg  string
+	repo string
+	tags []string
 }
 
 //
 func (s *image) ref() string {
-	return fmt.Sprintf("%s/%s", s.Repo, s.Path)
+	return fmt.Sprintf("%s/%s", s.reg, s.repo)
 }
 
 //
 func (s *image) refWithTags() string {
-	return fmt.Sprintf("%s/%s:%v", s.Repo, s.Path, s.Tags)
+	return fmt.Sprintf("%s/%s:%v", s.reg, s.repo, s.tags)
 }
 
 //
@@ -85,101 +86,124 @@ func newEnvClient() (*dockerClient, error) {
 }
 
 //
-func (dc *dockerClient) open() error {
-	var err error
+func (dc *dockerClient) open() (err error) {
 	if dc.client == nil {
 		if dc.env {
 			dc.client, err = client.NewEnvClient()
 		} else {
 			dc.client, err = client.NewClient(dc.host, dc.version, nil, nil)
 		}
 	}
-	return err
+	return
 }
 
 //
 func (dc *dockerClient) ping(attempts int, sleep time.Duration) (
-	types.Ping, error) {
-	var err error
+	res types.Ping, err error) {
+
 	for i := 1; ; i++ {
-		if res, err := dc.client.Ping(context.Background()); err == nil {
-			return res, err
+		if res, err = dc.client.Ping(context.Background()); err == nil {
+			return
 		}
 		if i >= attempts {
 			break
 		}
 		time.Sleep(sleep)
 	}
-	return types.Ping{},
-		fmt.Errorf(
-			"unsuccessfully pinged Docker server %d times, last error: %s",
-			attempts, err)
+
+	return types.Ping{}, fmt.Errorf(
+		"unsuccessfully pinged Docker server %d times, last error: %s",
+		attempts, err)
 }
 
 //
 func (dc *dockerClient) close() error {
-	var err error
 	if dc.client != nil {
-		err = dc.client.Close()
+		return dc.client.Close()
 	}
-	return err
+	return nil
 }
 
 //
-func (dc *dockerClient) listImages(ref string) ([]*image, error) {
+func (dc *dockerClient) listImages(ref string) (list []*image, err error) {
 
+	log.WithField("ref", ref).Debug("listing images")
 	imgs, err := dc.client.ImageList(
 		context.Background(), types.ImageListOptions{})
-	ret := []*image{}
-
-	if err == nil {
-		fRepo, fPath, fTag := util.SplitRef(ref)
-		for _, img := range imgs {
-			var i *image
-			for _, rt := range img.RepoTags {
-				matched, err := match(fRepo, fPath, fTag, rt)
-				if err != nil {
-					return ret, err
-				}
-				if matched {
-					repo, path, tag := util.SplitRef(rt)
-					if i == nil {
-						i = &image{
-							ID:   img.ID,
-							Repo: repo,
-							Path: path,
-						}
-						ret = append(ret, i)
+	if err != nil {
+		return
+	}
+
+	fReg, fRepo, tag := util.SplitRef(ref) // the filter components
+	name, dig := util.SplitTag(tag)
+	fTag := name
+	if dig != "" {
+		fTag = dig
+	}
+
+	for _, img := range imgs {
+
+		col := img.RepoTags // switch between the two lists depending on
+		if dig != "" {      // whether we have a digest as tag filter
+			col = img.RepoDigests
+		}
+
+		var i *image
+
+		for _, rt := range col {
+
+			if matched, err := match(fReg, fRepo, fTag, rt); err != nil {
+				return list, err
+
+			} else if matched {
+				rg, rp, tg := util.SplitRef(rt)
+				if i == nil {
+					i = &image{
+						id:   img.ID,
+						reg:  rg,
+						repo: rp,
 					}
-					if tag != "" {
-						i.Tags = append(i.Tags, tag)
+					list = append(list, i)
+				}
+				if tg != "" { // match has tag
+					if dig != "" {
+						// if we're using a digest as filter tag, we need to use
+						// the full tag from ref, since it may also contain a
+						// name, which would however be missing in tg; otherwise
+						// we take the tag as is from the match
+						tg = tag
 					}
+					i.tags = append(i.tags, tg)
 				}
 			}
 		}
 	}
 
-	return ret, err
+	return
 }
 
 //
-func match(filterRepo, filterPath, filterTag, ref string) (bool, error) {
+func match(filterReg, filterRepo, filterTag, ref string) (bool, error) {
+
+	if ref == "<none>:<none>" || ref == "<none>@<none>" {
+		return false, nil
+	}
 
-	filter := fmt.Sprintf("%s/%s", filterRepo, filterPath)
+	filter := fmt.Sprintf("%s/%s", filterReg, filterRepo)
 	filterCanon, err := reference.ParseAnyReference(filter)
 	if err != nil {
 		return false, fmt.Errorf("malformed ref in filter '%s', %v", filter, err)
 	}
-	filterRepo, filterPath, _ = util.SplitRef(filterCanon.String())
+	filterReg, filterRepo, _ = util.SplitRef(filterCanon.String())
 
 	refCanon, err := reference.ParseAnyReference(ref)
 	if err != nil {
 		return false, fmt.Errorf("malformed image ref '%s': %v", ref, err)
 	}
 
-	repo, path, tag := util.SplitRef(refCanon.String())
-	return (filterRepo == "" || filterRepo == repo) &&
-		(filterPath == "" || filterPath == path) &&
+	reg, repo, tag := util.SplitRef(refCanon.String())
+	return (filterReg == "" || filterReg == reg) &&
+		(filterRepo == "" || filterRepo == repo) &&
 		(filterTag == "" || filterTag == tag), nil
 }
 
@@ -216,8 +240,7 @@ func (dc *dockerClient) tagImage(source, target string) error {
 }
 
 //
-func (dc *dockerClient) handleLog(rc io.ReadCloser, err error,
-	verbose bool) error {
+func (dc *dockerClient) handleLog(rc io.ReadCloser, err error, verbose bool) error {
 
 	if err != nil {
 		return err