Should tags be signed in addition to digests?

[sudo-bmitch]

Most of the requirements and sample implementation work in nv2 has focused on signing the manifest by digest. That has a significant feature of image portability, allowing images to be retagged, or copied to another repository or registry, while maintaining the same signing data.

Do we also want the ability to also sign tags, indicating the acmerockets/heavylifter:1.5 currently has a specific sha digest? That would prevent a malicious actor from replacing the 1.5 tag reference with an older but still signed and trusted digest. If we were to sign tags, would it just be the 1.5 being signed or the full acmerockets/heavylifter:1.5. The full reference prevents portability but avoids the risk that another repository's image, e.g. acmerockets/bottlerocket:1.5, is sent as 1.5.

[marcofranssen]

Based on the JSON example shown here I would say there is no need for that, as the references to tags are attached to the digest already. Secondly most tags will shift from digest to digest based on your releasing strategy.

e.g. If I today release an image

acmerockets/heavylifter:1.5 I would also release acmerockets/heavylifter:1 and acmerockets/heavylifter:1.5.0 and acmerockets/heavylifter:latest.

Then on next release some of these tags will shift to the new release.

acmerockets/heavylifter:1.5 I would also release acmerockets/heavylifter:1 and acmerockets/heavylifter:1.5.1 and acmerockets/heavylifter:latest.

In general only the 1.5.0 tag didn't move to point to a new digest.

Not an expert though, but thought at least my reasoning would help in the discussion here.

[sudo-bmitch]

@marcofranssen just to verify, if you were to pull acmerockets/heavylifter:1.5.1, and a malicious registry (or MitM) returns the digest for acmerockets/heavylifter:0.5.9 which may have known vulnerabilities but not be removed from the repos, are you comfortable with notary saying you received a valid/signed image?

This also goes back to the TUF requirements of avoiding replay attacks where an old image can be presented as the current "v1" image.

[shizhMSFT]

Tags only are not suitable for signing since a tag can point to any manifests.

Instead, digests are suitable for signing since digest = Hash(manifest) cannot point other manifests with non-negligible probablity as long as Hash is a secure cryptographic hash function.

The tags are optionally signed for information only along with the digest.

[sudo-bmitch]

@shizhMSFT I wouldn't suggest that we stop signing digests, indeed we need that. However in notary v1 today, we only sign tags, with the indication that tag x currently points to digest y. From there, the manifest content itself is secure since changing that would modify the digest.

My concern if we only sign the digests is that users pulling a tag may receive a different image than intended. And if that different image digest was also signed, notary wouldn't indicate any MitM attack happened.

[marcofranssen]

@sudo-bmitch what about the sliding versions? my v2 tag that moves with any minor or patch release to a new digest? Would a new signature for that tag be sufficient to support this release strategy?

[sudo-bmitch]

@marcofranssen you would need to resign the v2 tag every time it points to a new digest, same as Notary v1 today.

[marcofranssen]

@sudo-bmitch agree, so what you are opting for is to make the tag signatures mandatory as opposed to optional to prevent the MitM example you gave earlier where my v2 tag all of a sudden points to a different digest.

[sudo-bmitch]

@marcofranssen Right now it's not even an option we specify. So this issue is asking if we "should" or "must" have tag signatures, or leave it undefined as it is today. I'm leaning towards any implementation "must" have the ability to sign tags, and then users could decide if they want to disable that feature on the client (enabled for security by default).

[mtrmac]

IMHO it absolutely must be possible for the user to rely on signatures to confirm that if the user asks for foo:1.5, they get foo:1.5 as signed by the original author regardless of what a registry does. If there are multiple images signed as foo:1.5, it may be acceptable to accept any one of them (the opposite would require a signature revocation mechanism).

[sudo-bmitch]

One option mentioned in today's call that I want to capture here is we can sign the full reference (registry, repository, and tag), and that clients can specify in their configuration that a local mirror is being used for an upstream registry. By having that client side mirror definition, a signature for upstream.reg/repo/image:1.5 when pulled from mirror.local/repo/image:1.5 would be considered valid by the client.

We may be able to get more complex with the mirror definition to handle more use cases, but I'm sure there will be exceptions where the image name and tag get changed. When that happens, there are a few options I can come up with:

• add another signature by a trusted party on the image copy
• configure the client to ignore tag signatures
• have some other way to configure the client to accept the different name (e.g. prompt with "image mirror.local/my/copy:busybox has a tag signature for docker.io/library/busybox:1.33, accept? [N/y]" for interactive users, and have a annotation indicating approved tags for a k8s environment)

Happy to consider other possibilities to bridge the gap between needing to know that the tag you pulled is correct without breaking the ability to copy images between repositories.

(cc @samuelkarp)

[mtrmac]

By having that client side mirror definition, a signature for upstream.reg/repo/image:1.5 when pulled from mirror.local/repo/image:1.5 would be considered valid by the client.

That’s what GitHub.com/containers/image implements, both as a transparent mirror mechanism where users refer to the upstream names ( https://github.com/containers/image/blob/master/docs/containers-registries.conf.5.md#remapping-and-mirroring-registries ) and mirrors are a transparent infrastructure concern, and as a signature policy mapping where users refer to the mirror locations but upstream names may be accepted ( https://github.com/containers/image/blob/master/docs/containers-policy.json.5.md#signedby remapIdentity); both can be combined to use three different values for the “end-user logical image name”, “signed identity”, and “physical registry location”.

This does mean that in order to introduce a new mirror/location, clients’ configuration must be updated to recognize/use it; OTOH it also allows all other parts of the ecosystem (CLI, K8s pod specs) to just use existing container image references as the only input without having to manually provide a trusted key / configuration stanza ID or other expression of what was intended / what image is acceptable.

[mnm678]

I think that the ability to sign tags in addition to digests should absolutely be part of this effort, so that we don't remove functionality that exists in Notary v1. In a similar vein, users should be able to access the most recent version of a tag, say v1.5, so that this tag always points to the latest intended image for v1.5, and not an old image (which may be insecure, or missing a new feature).

[joaopapereira]

As we talked in the slack message I do have a concern around resigning a particular tag. Like the following scenario:

1. Create a push image acmerockets/heavylifter@sha256:111
2. Tag the image with acmerockets/heavylifter:1.5.1
3. Sign the tag acmerockets/heavylifter:1.5.1 saying that the pointed image is: acmerockets/heavylifter@sha256:111
4. Create a new image acmerockets/heavylifter@sha256:222
5. Move the tag acmerockets/heavylifter:1.5.1 to the new image
6. Sign the tag acmerockets/heavylifter:1.5.1 saying that the pointed image is: acmerockets/heavylifter@sha256:222

Now we would have 2 images signed for the same tag. This could cause some sort of an issue because maybe we if the prior image as some sort of security vulnerability we might not want to keep that signature.

Should we only allow 1 signature per tag?

[mtrmac]

That wouldn’t help: some threat models include malicious/compromised registries / image delivery mechanisms, and once the old signature exists, we must assume the attacker keeps a copy; i.e. a registry-side enforcement of “single signature per tag” would not prevent an attacker from using the old signature with the old image (and a client-side enforcement does not work all that well in cloud-native deployments where the on-demand-scaled consumers don’t keep global state to notice existence the two signatures.)

The signature would have to somehow be revoked, using a mechanism that works even assuming a malicious network in the middle preventing communication; probably using a recent “this is still valid” timestamp that expires quickly, something like TUF — and the associated outages if the “this is still valid” timestamps can’t be generated or delivered. Revocation is a difficult trade-off vs. availability.

In practice, the signer can establish its own policy of not creating duplicate signatures (and forcing a tag update to 1.5.2 in such situations), without assuming that any registry/consumer enforces that. Sure, that’s an incomplete solution for users that want a “rolling” 1.x-latest tag. Do those occur in real-world production deployments all that often? I’d expect most CD pipelines to want to test and roll out a specific version.

[mnm678]

The signature would have to somehow be revoked, using a mechanism that works even assuming a malicious network in the middle preventing communication; probably using a recent “this is still valid” timestamp that expires quickly, something like TUF — and the associated outages if the “this is still valid” timestamps can’t be generated or delivered. Revocation is a difficult trade-off vs. availability.

Yes, this is one of the scenarios that TUF is designed to address that is not covered with just adding signatures to artifacts. The mechanism also works for other revocation scenarios.

[sudo-bmitch]

In practice, the signer can establish its own policy of not creating duplicate signatures (and forcing a tag update to 1.5.2 in such situations), without assuming that any registry/consumer enforces that. Sure, that’s an incomplete solution for users that want a “rolling” 1.x-latest tag. Do those occur in real-world production deployments all that often? I’d expect most CD pipelines to want to test and roll out a specific version.

While it's common to deploy a locally built image with a unique tag, when pulling in upstream images, the reverse of depending on a mutable tag is more common in my experience. This allows security updates to be applied without individually updating the usage of every upstream image+tag in every manifest and Dockerfile.

This is best solved with a TUF like solution (and can't be solved with only registry APIs since the registry could be malicious or MitM).

[joaopapereira]

Not sure but I feel like tag signing feels like a best effort action. That can give some confidence in the images being used but at the same time can be deceiving. Because even if a tag is signed it might be outdated or would require extra mechanisms to ensure that we are running the latest signed tag.

[sudo-bmitch]

I wouldn't call this best effort since we already solved this in notary v1 with TUF. The risk of running a stale tag was based on the timeout on the snapshot signature, which was kept intentionally short. But doing this in v2 on the registry instead of a dedicated notary server makes TUF a more difficult fit, liking moving that snapshot process to an external service and needing to resolve possible race conditions with multiple signers modifying the targets signature simultaneously. And if we support something like TUF, that will cause issues for the image and signature copying use cases.

We're left with three options:

1. Don't sign tags, client can resolve the tag to a digest and verify the digest was signed.
2. Sign tags, but provide no guarantee that the tag is current. Work is also needed to allow this tag signature to be portable.
3. Implement something like TUF, with possible issues for image copying and mirroring to offline registries.

[mnm678]

3. Implement something like TUF, with possible issues for image copying and mirroring to offline registries.

I opened a pr about copying TUF targets metadata between registries, and would appreciate any feedback. I think that solving image copying will be easier than re-inventing tag signing.

[SteveLasker]

I think that the ability to sign tags in addition to digests should absolutely be part of this effort

I think we've come to a consensus we need a way to say a digest:tag mapping is signed. And, if a newer digest:tag mapping is pushed, the registry can reflect that state.

It's also fair to say we need to protect from rollback

1. net-monitor:v1 (digest:a)
2. net-monitor:v1 (digest:b)
3. net-monitor:v1 (digest:a) must somehow be done in a secure and verifiable way. Hacking at the data of the registry to delete the v1:b mapping, causing a revert to v1:a should be protected from.

However, I'd suggest we also need to account for:

1. net-monitor@sha256:a is still valid and can be pulled by digest. It's not an invalid artifact, it's just not currently assigned to :v1
2. ACME Rockets has the choice when they import the update. If Wabbit-Networks publishes net-montior:v1 assigned to digest:b, it must be considered valid and safe to continue pulling net-monitor:v1 assigned to digest:a, within the ACME Rockets environment
3. ACME Rockets can choose to import the tag update.
4. ACME Rockets can also choose to rollback the tag update, when (not if) they get hit with a breaking update. They must not be locked to an update Wabbit-Networks has decided. The update could be a solar winds type of update the customer wants to rollback to, or it's simply an update that's valid for most customers, but breaks in the ACME Rockets environment.

How we achieve this flexibility is the interesting thing for consuming public content.

This solution must be flexible enough to support these scenarios, as the breaking change is the most common issue. Actual security attacks are few and far between. They're critical when they happen, and I think we all agree we must have a solution. The solution just can't lock a user to be forced to use the latest version of something that isn't actually compatible with their environment. And, we can't require a customer to change their deployment from net-monitor:v1, which now points to digest-b to a digest based deployment: net-monitor@sha256:a

Keeping this issue active, which reflects some of the challenges associated with Scenario 7.1

I'm torn on creating a new work-item/issue for tag signing, linking to this one, or re-title this one.
Thoughts?

[sudo-bmitch]

With 7.1 merged, and the Tag Signing design issue opened, I think this one is ready to be closed. Let move the rest of the discussion over to #63