Signature verification fails with: Storing signatures for docker tar files is not supported

[permanentdaylight]

I am hitting an issue where, when attempting to use skopeo to pull a signed image from Dockerhub and verify its signature, I'm getting Can not copy signatures: Storing signatures for docker tar files is not supported error.

I'm following this this guide on securing a Kubernetes production cluster.

(1) I'm creating a GPG keypair and then signing/pushing a local image to my dockerhub repo:

skopeo --debug copy --sign-by [email protected] docker-daemon:oakesonline/ubuntu:16.04 docker://docker.io/oakesonline/ubuntu:16.04

(2) I'm using the default sigstore settings as specified in `/etc/containers/registries.d/default.yaml':

[centos@ip-172-31-73-20 atomic]$ pwd
/var/lib/atomic
[centos@ip-172-31-73-20 atomic]$ tree
.
└── sigstore
    └── oakesonline
        └── ubuntu@sha256=e2bc06694c07f844dd1b90022e3a1d0be1c6bc2bb6b2036cc380c8c38f40aae3
            └── signature-1
3 directories, 1 file

(3) When I attempt to pull the image down from dockerhub and verify the signature I'm hitting the error.

[root@ip-172-31-73-20 containers]# skopeo --policy policy.json.STRICT --debug copy docker://docker.io/oakesonline/ubuntu:16.04 docker-daemon:oakesonline/ubuntu:sign-test
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000]  Using "default-docker" configuration
DEBU[0000]   Using file:///var/lib/atomic/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/docker.io
DEBU[0000] GET https://registry-1.docker.io/v2/
DEBU[0000] Ping https://registry-1.docker.io/v2/ err <nil>
DEBU[0000] Ping https://registry-1.docker.io/v2/ status 401
DEBU[0000] GET https://registry-1.docker.io/v2/oakesonline/ubuntu/manifests/16.04
DEBU[0000] IsRunningImageAllowed for image docker:docker.io/oakesonline/ubuntu:16.04
DEBU[0000]  Using transport "docker" specific policy section docker.io
DEBU[0000] Reading /var/lib/atomic/sigstore/oakesonline/ubuntu@sha256=e2bc06694c07f844dd1b90022e3a1d0be1c6bc2bb6b2036cc380c8c38f40aae3/signature-1
DEBU[0000] Reading /var/lib/atomic/sigstore/oakesonline/ubuntu@sha256=e2bc06694c07f844dd1b90022e3a1d0be1c6bc2bb6b2036cc380c8c38f40aae3/signature-2
DEBU[0000]  Requirement 0: allowed
DEBU[0000] Overall: allowed
Getting image source signatures
Checking if image destination supports signatures
DEBU[0000] docker-daemon: Closing tar stream to abort loading
FATA[0000] Can not copy signatures: Storing signatures for docker tar files is not supported

policy.json.STRICT

{
    "default": [
        {
            "type": "reject"
        }
    ],
    "transports":
        {
            "docker-daemon":
                {
                    "": [{"type":"insecureAcceptAnything"}]
                },
                "docker": {
                        "docker.io": [
                                {
                                        "type": "signedBy",
                                        "keyType": "GPGKeys",
                                        "keyPath": "/root/myproductionkey.gpg"
                                }
                        ]
                }
        }
}

Any help would be greatly appreciated. I'm new to skopeo and the image-signing ecosystem but I was under the impression this would work as it follows a similar flow to the linked article, but perhaps I'm mismatching Docker transport types or registry flow?

UPDATE:
I'm making a little progress as I was able to successfully enforce signing when using the containers-storage transport on my local machine. In contrast, I received errors when using using both the docker-daemon and oci transports for storing the images. Is signing supported on all transports?

[rhatdan]

@mtrmac PTAL

[mtrmac]

Thanks for your report.

Neither Docker daemon nor OCI supports storing the signatures, and skopeo copy refuses to just silently drop them.

You can use skopeo copy --remove-signatures to make the copy anyway; the signatures will be still read and policy.json will still be enforced, they just won’t be written into the destination.

Probably more conveniently, docker pull, when using a build from https://github.com/projectatomic/docker (e.g. a RHEL or Fedora one), natively includes policy.json support. Or you can perhaps use atomic pull, which wraps skopeo the way you are calling it, but includes the --remove-signatures automatically.

And, or course, as you have discovered, containers-storage: does record the signatures, so podman pull will do the right thing automatically.

[permanentdaylight]

@mtrmac I appreciate the detailed response; I'll explore some of these other signing options you mentioned. Thanks again.

[mtrmac]

Thanks; closing, then, but feel free to reopen if anything else comes up.