[19.03] gnupg: change default keyserver to non-SKS

[alyssais]

Motivation for this change

Backport of #63952. I expect this to be controversial since it is a breaking change, and I know @Ekleog disagrees, so I’m opening this for discussion and to allow the release managers to make the call.

Personally, I think this is the right thing to do. keys.openpgp.org may not support all of SKS’s features, but it’s the only solid way to ensure our users are protected against their GnuPG installations becoming unusable. We could set import-clean by default, which protects against the attack iff the attacker has not managed to already get their own key into the user’s key ring, but doing that isn’t exactly difficult, especially if they enable auto-key-retrieve.

To summarise our options:

• Do nothing. Almost any interaction with a key server can potentially render a user’s GnuPG installation unusable, requiring surgery or deleting the keyring to be usable again.
• Enable import-clean by default. Mitigates the attack in some cirmcumstances, but not all by far.
• Switch the default keyserver. Protects against the attack unless the user explicitly configures otherwise. Means that some keyserver features (UID fetching, UID revocation) are no longer available.

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Assured whether relevant documentation is up to date
• Fits CONTRIBUTING.md.

---

[lukateras]

Some context. Current default keyserver (SKS pool) allows for certificate poisoning attacks which can effectively brick user's GnuPG installation. Attack is based on adding tens of thousands of signatures to the key, at which point, when user ends up importing the key (including unwillingly, via gpg --recv) GnuPG operations may require up to 10 minutes of CPU time: https://dev.gnupg.org/T4592#127389

The problem lies with SKS not having any rate limiting and exacerbated by SKS accepting arbitrary OpenPGP packets, including cryptographically invalid packets, including non RFC compliant packets. There are a few similar attacks documented on SKS BitBucket, and there are likely more. SKS is not being maintained at this point, so this is unlikely to get fixed.

https://keys.openpgp.org is more resistant to these kinds of attacks, at cost of losing capability to synchronize between servers. It is a Hagrid keyserver instance that is being run by Sequoia, OpenKeychain and Enigmail developers.

[lukateras]

This is a breaking change, but given circumstances and severity, I believe it should be backported.

[Ekleog]

Backwards-incompatibilities:

• The content base of keys.openpgp.org is running from a dump of sks-keyservers.org, which therefore doesn't include the latest additions
• It becomes impossible to do --recv-key with anything other than a fingerprint or a verified email address, which will likely, when coupled with the fact there are currently ~2k verified email addresses, kill a lot of tutorials and workflows when using gpg --recv-key, as well as probably some ill-written scripts
• Third-party signatures are no longer fetched along with a key, which means that web-of-trust validation becomes impossible, which breaks all the remaining tutorials and workflows, and most likely lots of scripts that rely on auto-key-locate or similar to fetch keys and then verify them by WoT
• UIDs are stripped and only verified UIDs are downloadable, which means human usage becomes really really weird (eg. I guess it makes for a broken enigmail UI for all but the 2k verified email addresses, in addition to the no-WoT issue meaning no key would ever show as validated except the ones that had been downloaded before this change)

Seriousness of the issue:

• The issue has been known and discussed for years. For instance, I have references to trollwot since early 2016 in my inbox from [email protected] (and my email archives from before late 2015 are not easily searchable)
• The issue is a denial-of-service on gnupg installations that download a poisoned key from sks-keyservers, that is coupled with an issue that sks-keyservers let anyone poison any key
• The impact of the denial-of-service is that gnupg is going to basically fail every time it attempts to use the key, which is going to be pretty often though not always
• The denial-of-service condition can be recovered from by:
  • Deleting the affected key(s) (currently 3 known ones) from the keyring and re-importing them from a known-good source, or
  • Scrubbing the keys from untrusted signatures (which is actually much harder to do, though not impossible)

Why the issue is making noise now, while it had been more or less ignored in the wider world until now (though this issue is one of those that prompted development of the software keys.openpgp.org runs and it has been discussed lots of time):

• Now one person has started actually exploiting this well-known issue
• It directly affects a handful of human people (3 as far as I know, right now), which are therefore naturally emotionally affected

Other mitigations:

• As far as I know, import-clean is the only other mitigation. It should protect against most non-targeted attacks, but would not protect against a targeted attack.

So, with all of the above facts now known, my opinion is that this issue is not actually that bad. And that, actually, the fix would, for the stable channel, cause more damage (by breaking all scripts that rely on talking to keyservers and/or using the WoT) than the initial issue causes damage.

On the other hand, turning on import-clean still has a small likelihood of breaking scripts, and does not protect against targeted attacks. But it does protect against non-targeted attacks, and its likelihood of breaking scripts stays really low.

So I think the path we should choose forward should be:

1. Enable import-clean by default
2. Send out a security advisory telling people that we are not forcing this change upon them for the above-mentioned backwards-incompatibility reasons, but that they should fix their systems and either set their keyserver manually or use the patched GnuPG that would be available under another attribute name if they have multiple users and want to generically change the default

[fpletz]

Please do not merge until #64256 is resolved and the fix can also be backported.

[dkg]

• The content base of keys.openpgp.org is running from a dump of sks-keyservers.org, which therefore doesn't include the latest additions

agreed. please help make progress on fixing this

• UIDs are stripped and only verified UIDs are downloadable, which means human usage becomes really really weird (eg. I guess it makes for a broken enigmail UI for all but the 2k verified email addresses, in addition to the no-WoT issue meaning no key would ever show as validated except the ones that had been downloaded before this change)

What human usage are you concerned about? The current human usage i'm seeing is that people are finding themselves locked out of their e-mail programs because "mail.app crashes" (probably not relevant for nixos) or "thunderbird is unresponsive" or "sending mail takes forever". I've even heard of people deleting their mail stores to try to debug this problem.

• The issue has been known and discussed for years. For instance, I have references to trollwot since early 2016 in my inbox from [email protected] (and my email archives from before late 2015 are not easily searchable)

trollwot's git repo dates to 2013. the fact that it is not recent does not mean it is not serious. The time to deal with it as a low-urgency fix was 2013, and we as a community failed to do that. Now that it is actively being exploited it is indeed a high-urgency situation.

• The impact of the denial-of-service is that gnupg is going to basically fail every time it attempts to use the key, which is going to be pretty often though not always

gpg also fails when you are using other keys, or even when you are just trying to list the set of keys in your keyring. please see upstream performance reports.

• Deleting the affected key(s) (currently 3 known ones) from the keyring and re-importing them from a known-good source, or

There are definitely more than 3 keys currently flooded (i know of 6 currently) and it is trivial for anyone to flood all of them.

• It directly affects a handful of human people (3 as far as I know, right now), which are therefore naturally emotionally affected

As someone whose certificate was flooded, I am indeed emotionally affected. I wrote about the emotional effect. But the people whose keyrings contain the flooded certs are the ones who are directly affected, not just the people to whom those certificates belong. Any keyring that contains the certs become unusable.

So, with all of the above facts now known, my opinion is that this issue is not actually that bad. And that, actually, the fix would, for the stable channel, cause more damage (by breaking all scripts that rely on talking to keyservers and/or using the WoT) than the initial issue causes damage.

If those scripts are currently used in any critical systems, those scripts are subject to breakage by an arbitrary attacker at any time, and once flooded, the attack will be persistent because even deleting the known key from the keyring can take 15 minutes or more on flooded installations. A clean break that avoids permanent ingestion of flooded certificates seems healthier to me, though I can understand seeing it the other way around too.

On the other hand, turning on import-clean still has a small likelihood of breaking scripts, and does not protect against targeted attacks. But it does protect against non-targeted attacks, and its likelihood of breaking scripts stays really low.

I agree that enabling import-clean by default is a reasonable step to take. I've suggested it to upstream too. In general, please consider pushing your own recommendations upstream rather than keeping them local to NixOS.

2. Send out a security advisory telling people that we are not forcing this change upon them for the above-mentioned backwards-incompatibility reasons, but that they should fix their systems and either set their keyserver manually or use the patched GnuPG that would be available under another attribute name if they have multiple users and want to generically change the default

Encouraging people to create custom configuration files that will linger long after they have forgotten about the problem doesn't seem like a great idea. I don't understand NixOS' "attribute names" so i can't comment on that part.

[alyssais]

Please do not merge until #64256 is resolved and the fix can also be backported.

Sorted.

[Ekleog]

Thank you for your precisions! For import-clean, I took the idea from the ML, and assumed it was being tracked upstream already -- thank you for pointing out that it wasn't and actually starting tracking it.

As for NixOS' “attribute names,” it'd be basically equivalent to providing both the current gnupg as well as a gnupg-patched that would include a patch that makes keys.openpgp.org the default; so that by default things stay stable but simple user action could replace one with the other.

I think that we now have as much information as we could have, and the last thing we need to do is choose, for the last ~4-5 months 19.03 will live, between Scylla and Charybdis.

@lheckemann @samueldr Do you have enough information to make a choice here? I think it'll be hard to gather a consensus around the one best option, but hope everyone will agree that both paths are about as bad, with both pros and cons for each.

If other people want to give an opinion on the matter without not-yet-mentioned arguments, I suggest using emojis on this comment:

• ❤️ for “let's change the default keyserver”
• 🚀 for “let's use import-clean”
• 🎉 for “let's use self-sigs-only”
• 👀 for “let's do nothing and wait out the ~4-5 months with lots of hope” (well, this path also exists and is going to be more “stable” than the other two so I can't not put it here, but I must say it doesn't really play in the same league as the two other choices to me)

[alyssais]

Agree with all of dkg's points. This is only going to get worse, and inaction puts our users at risk at some random future time when somebody tries to attack them. Mitigating the issue will at least give our users clear steps to take, and we can communicate, rather than allowing a third party to disrupt them.

"mail.app crashes" (probably not relevant for nixos)

FWIW Nixpkgs also supports macOS, so we should are concerned about this.

I don't understand NixOS' "attribute names" so i can't comment on that part.

Think of it like a package name in other systems. It's how you refer to a package to install it.

[wiktor-k]

@Ekleog, one more thing to keep an eye on is --keyserver-options self-sigs-only.

[Ekleog]

@wiktor-k In my opinion, if we think of doing self-sigs-only, then we might just as well switch to keys.openpgp.org, as self-sigs-only has the same drawbacks as keys.openpgp.org except even stronger (as it doesn't even have said ~2k keys keys.openpgp.org has) -- in exchange it does have the latest up-to-date sks-keyservers dump… hmm… well, adding that one to the poll, as 🎉 :)

[wiktor-k]

@Ekleog, self-sigs-only as far as I understand will drop all non-self-signatures from a key. So compared to keys.openpgp.org you can import any key and you'll get full User IDs, no verification necessary so as many keys as SKS has you can import them all and they'll be usable (currently keys with no User IDs aren't nicely supported in GnuPG).

The real drawback of self-sigs-only is no Web of Trust. This is an issue for some parties (e.g. Gentoo that has their own CA).

I'm not pushing for it, just wanted to share the option.

(For some closed circles and when one controls the domain Web Key Directory is a fine middle-ground, keys are fetched over secure connection and cannot be spammed. Gentoo and Debian already publish developer key's via WKD, is NixOS planning to? :) ).

[Ekleog]

@wiktor-k Oh, I was thinking it did not import User IDs either. Thank you for the clarification!

(as for WKD, NixOS doesn't provide developers with an @nixos.org email address, so AFAIR NixOS can't provide WKD even if it wanted to :))

[wiktor-k]

(@Ekleog, got it, for now I'll need to rely on the "centralized" [Member] badge) :)

[lukateras]

Encouraging people to create custom configuration files that will linger long after they have forgotten about the problem doesn't seem like a great idea. I don't understand NixOS' "attribute names" so i can't comment on that part.

@dkg: in this context it means creating a separate GnuPG package with this fix.

There are definitely more than 3 keys currently flooded (i know of 6 currently) and it is trivial for anyone to flood all of them.

Can confirm: my key has also been flooded (likely because of my involvement in SKS issues similar to this one). I don't use it for email (I have a separate Autocrypt key for that), but if anyone tries to fetch it, say to check any of my commit signatures in this repo, their GnuPG install will be broken (and recovery time will depend on how much my key has been flooded: I don't know the exact size).

---

• I want to reiterate what @dkg touched upon: there is nothing stopping someone poisoning every single popular key in this manner, which will break GnuPG install of anyone relying on human usage @Ekleog mentioned. It would be wrong to expect no one will do just that.
• If any public facing services use GnuPG and fetch arbitrary keys from SKS (for example, user's keys), this will be a major exploitable DoS vulnerability for them.
• There are other (albeit related) issues/attack vectors that may make SKS inoperable; see Bitbucket issues 57 (partly fixed on the client for some types of OpenPGP packets, but still exploitable), 60, 61, 63. import-clean or even self-sigs-only won't help with any of these.

[lukateras]

5 days in, 2 ❤️ and 1 🚀. I'd like to go ahead and merge this as-is on Wednesday unless there are any concerns left over doing so or vote result changes.

Backported in 46dc464

[lukateras]

cc @fpletz @grahamc

[Ekleog]

@lheckemann @samueldr Could you confirm whether you're OK or not with merging this? It sounds like a complex enough decision to warrant some comment from one of you. All the information needed for deciding should be available in the comment at #63964 (comment) and the post by @dkg just below.

[lheckemann]

I'm unsure, though that isn't a -1. Do we have some established procedure for announcing necessary breaking changes in stable versions?

[Ekleog]

For the time being, none that I know of, apart from an announcement on the discourse. (And, similarly, we have no way that I know of to announce a security vulnerability we can't properly patch on our side, apart from an announcement on the discourse.)

Aside: I guess a good first step to solve this other issue might be to open a moderated discourse category and add a paragraph in the manual to tell people to subscribe to it, so that we would at least have a way of pushing information to end-users, until we figure out some way to push information at nixos-rebuild time or similar.

[vcunat]

Well, I didn't study details of this, but at least I pushed e8e1eec which is claimed to mitigate the problem (and I see no controversiality about that bump).

[Ekleog]

@vcunat The change in e8e1eec is basically import-clean + self-sigs-only, which in some way is pretty rough for backwards-compatibility too (options 🚀 plus 🎉 listed in my post above). Now, it's the choice upstream decided was best for their stable line, so… meh.

[Ekleog]

Just came across https://bugs.archlinux.org/task/63147, to make the kind of backwards-compatibility breakage I'm speaking of more explicit. (Now, at least in nixpkgs we don't rely on the keyservers, so we wouldn't be breaking all NixOS machines -- “just” the scripts that would behave like pacman does)

[vcunat]

Ah, I see it's not as straightforward as I originally thought.

[veprbl]

The 19.09 is out. Is this still relevant?

[alyssais]

It is not. This change is the default in stable NixOS starting from 19.09.