Copy statistics

[estahn]

Would it be possible to generate and return statistics on how much data was copied?

[mtrmac]

The containers/image/copy.Image library allows the caller to configure a Progress receiver; that could allow collecting/computing some statistics (though it probably could be extended quite a bit).

As far as the Skopeo CLI, the progress bars do contain some size information, but it’s human-focused, not structured for parsing/summarization, and I’m not sure how to combine the two well. Do you have a specific proposal in mind?

[estahn]

@mtrmac Just some background, I'm using skopeo in k8s-image-swapper and would like to provide statistics via prometheus, e.g. primarily how much data was copied, how many layers, the total size of the image, etc.

I would imagine providing an output format parameter similar to kubectl would be useful, e.g. skopeo copy --output=json ....

[mtrmac]

That could work, though figuring out a long-term stable JSON format might not be quite trivial, especially for in-progress updates where copy.Image might not provide enough information right now.

Also perhaps compare #658 .

Would you be interested in preparing a PR?

[estahn]

@mtrmac I'm trying to implement copy.Image to see how the ReportWriter works.

1. Looking at the snippet below, using policy := signature.DefaultPolicy(nil) will result in a sigfault. Any ideas why?
2. Does the ReportWriter uses structured information or is it just a stream of string I would need to parse? Given Please provide a basic progress / transfer speed output ("CI compatible") on copy #658 having some sort of stream with the context of metrics and various aggregated views might be useful.

Snippet:

package main

import (
	"context"
	"os"

	"github.com/containers/image/v5/copy"
	"github.com/containers/image/v5/signature"
	"github.com/containers/image/v5/transports/alltransports"
	"github.com/containers/image/v5/types"
	"github.com/davecgh/go-spew/spew"
)

func main() {
	ctx := context.TODO()
	policy := &signature.Policy{Default: []signature.PolicyRequirement{signature.NewPRInsecureAcceptAnything()}}
	//policy, err := signature.DefaultPolicy(nil)
	policyCtx, _ := signature.NewPolicyContext(policy)
	defer policyCtx.Destroy()

	srcRef, _ := alltransports.ParseImageName("docker://docker.io/library/busybox:latest")
	destRef, _ := alltransports.ParseImageName("dir:/tmp/images")

    spew.Dump(srcRef, destRef)
	spew.Dump(srcRef.Transport().Name())
	spew.Dump(destRef.Transport().Name())
	spew.Dump(policyCtx.Policy)

    srcCtx := &types.SystemContext{
		DockerRegistryUserAgent: "k8s-image-swapper/1.1.0",
	}
	destCtx := &types.SystemContext{
		DockerRegistryUserAgent: "k8s-image-swapper/1.1.0",
	}


	opts := &copy.Options{
		ReportWriter:       os.Stdout,
		SourceCtx:          srcCtx,
		DestinationCtx:     destCtx,
		ImageListSelection: copy.CopyAllImages,
	}

	image, err := copy.Image(ctx, policyCtx, destRef, srcRef, opts)

	spew.Dump(image, err)
}

[mtrmac]

@mtrmac I'm trying to implement copy.Image to see how the ReportWriter works.

I’d expect this should end up integrated in cmd/skopeo/copy.go, which does call copy.Image already — but experimentation and feedback on the API is of course welcome, and initial experimentation could well be simpler without the whole cmd/skopeo infrastructure.

---

1. Looking at the snippet below, using policy := signature.DefaultPolicy(nil) will result in a sigfault. Any ideas why?

Are you saying DefaultPolicy segfaults? I can’t immediately see how that can happen, that’s a bug we should fix. Do you have a backtrace?

A possible crash path is that if DefaultPolicy fails with an error, it returns a nil policy, and using that nil crashes; in that case the caller of DefaultPolicy should be handling the error before using the returned policy object.

---

2. Does the ReportWriter uses structured information or is it just a stream of string I would need to parse? Given Please provide a basic progress / transfer speed output ("CI compatible") on copy #658 having some sort of stream with the context of metrics and various aggregated views might be useful.

ReportWriter is not what this feature would be built around; in fact we might want to silence it to io.Discard so that the standard output, in the --output=json mode, doesn’t intersperse JSON and non-JSON output. (Or, alternatively, one of the two would go to a different file descriptor / named file.)

See copy.Options.ProgressInterval and copy.Options.Progress; https://github.com/cri-o/cri-o/blob/c8bbae9858a084f4244cbd3bb38852d29fc0466b/server/image_pull.go#L173-L211 is an example consumer, just to show the basic structure.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.