Unable to pull and verify images signed with a GPG *subkey* ("None of the signatures were accepted") #16406
Comments
|
Yeah… the system is designed to only accept signatures by a specific set of key identities (and that enforcement exists even if the user provides a single public key file, where we extract the identities from the keys themselves). And GPGME, which we use, awkwardly returns the primary fingerprint on import, but the subkey fingerprint on signature verification. It seems possible to support subkeys in principle (it’s possible to list the subkeys of a key), but that would require manually dealing with expiration/revocation and the like. And that extra key inspection roundtrip would show up on every single signature verification; IIRC https://dev.gnupg.org/T3380 is still not fixed upstream, so that would be a whole extra second. So at least from a quick look, this seems non-trivial to handle. Or maybe I’m missing some obvious solution. I’m also not immediately clear on the semantics of signing subkeys … if a signing subkey to be revoked, what does that mean for signature validity? We get into the realm of third-party timestamping servers, and the like, none of which really exist in the ecosystem or existing container signing workflows. OTOH if a signing subkey can’t effectively be revoked, there’s much less benefit to it. |
|
Thanks for the quick responses. I'm not sure how third party servers are involved here. A signature verification is a local operation. You should be able to extract from gpgme if a subkey is valid, expired or revoked. But all of this should be local data that you can get from GPG. The simplest is just to never treat a signature as valid if the key has been revoked (even if the revocation happened after the signature was made).
Somewhat unrelated. But I would greatly appreciate if the podman policy would allow specifying key fingerprint instead of |
|
As an example, if you sign a commit in git with a valid signing subkey (something we do all the time), and then you revoke the subkey, then git will say this: And the exit code from that command is So I think the logic for podman could be somewhat simple: Find the (sub)key that signed the container image. Check if it is valid (!expired && !revoked && !disabled && !invalid) and then validate the signature with it? |
|
(Warning: first impressions without really spending enough time to consider the trade-offs.)
Whether a signature is valid depends on the time the signature was made, and under the assumption that the revoked subkey was compromised, the signature itself is not a trusted source of timestamp information.
Yeah, but then revoking the subkey, and distributing that updated public key, immediately breaks all pulls of the previously-signed images, if they are copied through some mirror/distribution system. So it seems to me that, for consumers, “how to deal with revoked subkeys” ends up being a separate policy dimension that users would want to configure, so that they can smoothly transition over without downtime. Or maybe they would need to carefully manage the without-revocation and with-revocation public key files, and make sure that the “real-world” revocation status never leaks onto a computer that is not ready to work with it. And from the vendor’s point of view (for container-image-signing-only keys?) there seems to be little benefit to the subkey revocation — if the vendor needs to re-sign everything, and tell customers to make specific configuration choices to deal with the transition, the vendor might just as well generate a completely new key. Having new key would already allow a smooth transition without another configuration option — the policy already supports providing a trusted keyring, so users can choose which of the two top-level keys to trust, without having to introduce a new mechanism. I guess all of this calculation changes if the keys primarily identify humans, as opposed to using single-purpose machine-managed keys (e.g. “this key exists only to sign images produced by $vendor’s build system and approved for production usage”). The single-purpose keys are closer to the originally anticipated usage; humans are important :) but larger companies tend to avoid single points of failure like that.
Very unrelated — if you want this, please file this separately. It would not be too hard to do, but ~90% of the work would be ensuring adequate test coverage because it doubles the test matrix of …some… test scope. Meanwhile, the sigstore configuration format is where we expect to be spending most new effort… |
I'd argue that this is not true. As I referenced above: Git treats all signatures made by a key as invalid if it knows the key is revoked, even if it was revoked after the signature was made. Time is irrelevant. Many people who use GPG and employ revocation generate the signed revocation message directly when they generate the keys. Then they keep the revocation file protected and release it to the world if the key is ever compromised. That suddenly makes everything that key every signed invalid. |
|
Yeah, but Git does not enforce signature validity in a comparable way. A Git commit with a later-revoked signature continues to exist in the history forever. A signed container image with a revoked signature + a random need to actually pull it (e.g. because a computer went down and the workload is being migrated to another computer) = application downtime until a human intervenes. |
|
Yes indeed. If you have learned that a key is revoked, you probably want to immediately stop trusting anything it ever signed. If someone revokes their key because the key has been compromised, no signature can be trusted, since it can't be proven if the signature was made before or after it was compromised. The person revoking the subkey should create a new subkey that is signed with the master key and re-sign any data that is still important to other people. |
|
If a user wants to pull an image signed with a revoked key and they truly don't care. They can disable the signature checking policy for this image, or they can modify the public key file to not contain the revocation, making it still look valid. If that's their threat model, they can do that. Pulling in the revocation in the first place was their own decision. |
|
Btw, how is this different for subkeys compared to main keys? I don't see how a revoked main key would work any different compared to a revoked subkey 🤔 |
Maybe I misunderstand here, but this is exactly what subkeys are for. The main key should be hidden, and only the subkey exposed to the machine that signs images. If a subkey is compromised the maintainer can revoke it and generate a new subkey that is signed with the same main key. They then distribute the updated key and any consumer of this can see that the subkey changed but is signed with the already trusted main key. This way they don't need to re-evaluate the trust in the main key, which would be the case if the maintainer created a completely new key with no ties to the old key. If the maintainer does this, then the consumers will need to find a secure way to obtain the fingerprint from the vendor and manually verify the new key material against it. |
|
(That’s just not how security works in practice. Even critical known-to-be-exploited vulnerabilities are very rarely a reason to take the whole application down; users frequently value the application availability too much. Hence, change windows, testing deployments, staged roll-outs… I’m not saying that accepting revoked signatures should be the default, just that many users would probably want to explicitly control it. So that’s extra complexity.)
Exactly. From the image consumer’s point of view, what’s the difference between distributing a new file with the same primary key but a subkey change, and between just distributing a completely new primary key? It ends up with the same deployment breakage AFAICS; the only difference is that we already have features for transitioning between primary keys. It seems to me that subkeys only help if the vendor can distribute images signed with the new subkey before revoking the old subkey. Realistically, any kind of key compromise is going to involve manual human-to-human communication (or perhaps even phone calls). The ”simple” part of the “simple signing” name exactly refers to having no ambition to implement any kind of automated key compromise recovery (contra Notary / TUF), because the complexity rises so quickly. |
Note that container signatures don’t carry the public key along in the signature blob (and EDIT if a signature can somehow contain a subkey update, at this point I think I’d consider that a vulnerability). So signature consumers need to manually decide to distribute an updated public key record either way, and that will involve some human having to think whether the public key file update is legitimate. |
If they get a new primary key they need a way to verify its authenticity. Usually by obtaining the fingerprint from some known good source and comparing it. They don't need to do this if only the subkey changes, because then they keep the trust in the primary key, which remains the same. Which is exactly what subkeys are for. The owner of the primary key whose subkey has been compromised can distribute the subkey revocation along with a new subkey, and the downstream consumer can have 100% automated systems for importing it. Because there is nothing new that needs to be verified manually. |
They already can, can't they? Well, if this bug this issue is about didn't exist that is. If a key is compromised the procedure from the vendor would be:
Interesting. I know nothing about this. Could you please point me to documentation around this? I have had a really hard time on finding good documentation on |
I'm fine with both of these. All I want is the possibility to use subkey signatures at all. We don't want to expose our master keys outside air gapped systems. So the signing servers can't have it. |
The The actual timing of the transition would need to be managed explicitly, by distributing policy / file updates; and those updates would need to be manually prepared. But it’s at least possible now.
I’m not really happy about introducing extra complexity with little tangible benefit. Nor about the at-least-one-second delay currently contemplated. It’s definitely worth tracking, and seeing if we can find an elegant solution. |
No, not really. If you have primary key
The tangible benefit is that it's pretty standard to use subkeys and not primary keys for anything (there are many sources on this, but here is one). And you don't need to expose the primary key to the servers signing images. Also, the tangible benefit is that it becomes usable for people who already have keys. At my organization we have a publicly known GPG key that has been around since 2016. And by default if I use So one potential solution that is not optimal, but at least a step in the right direction: Make |
That’s not what I think usually happens. The web of trust is not really relevant. Administrators get a vendor’s public key file somehow — most likely from a web site telling them how to actually use the containers they are choosing to deploy, with the hundreds of CAs that could possibly compromise that HTTPS connection — and install it on the target computer as is. They have no reason to If a new public key file is ever distributed to production systems, that’s almost certainly because some human made a decision to do so. I think a minimal target to aim for here is to accept the subkey signatures iff GPG itself accepts them. And punt on the user control of revocation behavior, at least for now. I don’t think that’s particularly useful but I very much agree that |
|
I guess our solution at the moment will be to create a brand new GPG key (without subkeys), sign it with our main key and distribute it to all employees. We don't really use containers to deploy anything, but rather distribute trusted development environments to all programmers. If a key is ever compromised, we immediately want everyone to stop using the containers signed with that key. However, I think the bug at hand should be fixed still. There are lots of GPG keys in circulation that has subkeys, as it's a recommended practice in general for GPG. All our current keys (company main key as well as all developers' own keys for emails and git signing) are at the moment not possible to use with |
|
I don't know how you like to track issues. I have stated what the issue for us is, and we don't have the expertise to provide a patch. So I'm not sure there is much else I can contribute to this thread. I still think the sane thing to do here is to allow signing images with subkeys and verify them. That's what every other GPG signing integration I have come across is doing. |
|
I don’t have an opinion on whether this is a bug or a RFE. To help the Podman bug tracking decisions: It’s something that should be fixed, and I think worth tracking; but it also requires more GPG/GPGME expertise than I think is readily available for a quick fix, and in the next few months I’ll be working on sigstore, not on this issue. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
@vrothberg Care to handle this one? |
Unless we deprioritize other items, I can't. @mtrmac, do you have cycles? |
|
I’m afraid I don’t. |
|
A friendly reminder that this issue had no activity for 30 days. |
|
Hi |
/kind bug
Description
I'm unable to pull and verify an image signed with a GPG subkey, but it works fine if the image was signed directly with the main/master key. It is good practice to work with subkeys and only use the main GPG key for signing the subkeys. But this recommended workflow does not currently work with podman for me. I have tried this with both ed25519 and rsa4096 keys, same issue. I run podman in rootless mode and have not tried this in root mode.
Steps to reproduce the issue:
~/.config/containers/registries.d/default.yaml:~/.config/containers/policy.json:{ "default": [{ "type": "insecureAcceptAnything" }], "transports": { "docker": { "ghcr.io": [ { "type": "signedBy", "keyType": "GPGKeys", "keyPath": "/home/faern/sandbox/key.pub" } ] } } }GPG key:
Dockerfile(probably reproducible with any type of image you sign):Commands:
podman build . --sign-by=8716EC50A50A9744EF8F573DE163D46385F9421A -t ghcr.io/faern/faern-sign-testpodman push --sign-by=8716EC50A50A9744EF8F573DE163D46385F9421A ghcr.io/faern/faern-sign-testpodman rmi ghcr.io/faern/faern-sign-testpodman pull --log-level debug ghcr.io/faern/faern-sign-testDescribe the results you received:
Output of the
podman pullcommand above:73C091A52C48057289DC3AB98981BCE19E42BD72is the fingerprint of the signing subkey.However, the signature files that
podman pushproduced are valid:If I remove the subkey from the GPG key:
... And re-run the commands above. Then it works fine. Output of the
podman pull:Describe the results you expected:
I would expect to be able to verify images signed with GPG subkeys. This works fine with regular gpg (
gpg --sign+gpg --verify) and is often the recommended setup of your keys.Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info:GnuPG version:
Package info (e.g. output of
rpm -q podmanorapt list podmanorbrew info podman):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
I have tested with 4.2.1 as it's the newest version in my distro. But I can't see anything about sigstores or GPG in the release notes for the newer versions. Also nothing on this reported in the issue tracker. So I doubt it's fixed in 4.3.0.
The troubleshooting guide has nothing on sigstores or GPG.
Additional environment details (AWS, VirtualBox, physical, etc.):
Physical machines with Fedora 36.
The text was updated successfully, but these errors were encountered: