acme/lego causes account rate limits

[rkoe]

Describe the bug
When upgrading to 20.03, the change from simp_le to lego causes

• all certificates to be re-created (which is not nice, since the old certificates would still work)
• if more than 10 certificates are used on a server, some certificates cannot re-created, because lego creates a new account per certificate an so runs into the account rate limit (max. 10 accounts in 3h per IP) -- and so the site-encryption breaks for servers with > 10 certificates.

According to #85152 this should be fixed -- but it isn't.

To Reproduce
Steps to reproduce the behavior:

1. Use a server with e.g. NixOS 19.09 and > 10 certificates.
2. Upgrade to 20.03
3. The upgrade will result in failing acme-services.

Metadata

• system: "x86_64-linux"
• host os: Linux 4.19.114, NixOS, 20.03.1445.95b9c99f6d0 (Markhor)
• multi-user?: yes
• sandbox: yes
• version: nix-env (Nix) 2.3.4
• channels(root): "nixos-20.03.1445.95b9c99f6d0"
• nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

[arianvp]

all certificates to be re-created (which is not nice, since the old certificates would still work)

Point one is not a bug. It's explicitly mentioned in the release notes as an unavoidable tradeoff.

Certificates will be regenerated on activation, no account or certificate will be migrated from simp-le. In particular private keys will not be preserved. However, the credentials for simp-le are preserved and thus it is possible to roll back to previous versions without breaking certificate generation. Note also that in contrary to simp-le a new private key is recreated at each renewal by default, which can have consequences if you embed your public key in apps.

[arianvp]

@rkoe could you show /var/lib/acme/.lego on your server? I cant reproduce what you're describing. I only have one account for all the certificates I manage:

[root@arianvp:~]# tree /var/lib/acme/.lego/
/var/lib/acme/.lego/
├── accounts
│   └── acme-v02.api.letsencrypt.org
│       └── [email protected]
│           ├── account.json
│           └── keys
│               └── [email protected]
├── arianvp.me
│   ├── accounts -> ../accounts
│   └── certificates
│       ├── arianvp.me.crt
│       ├── arianvp.me.issuer.crt
│       ├── arianvp.me.json
│       └── arianvp.me.key
└── techstock.photos
    ├── accounts -> ../accounts
    └── certificates
        ├── techstock.photos.crt
        ├── techstock.photos.issuer.crt
        ├── techstock.photos.json
        └── techstock.photos.key

10 directories, 10 files

You might be running into a certificate rate limiting issue instead. The 19.09 version of the module was vulnerable to this too and is not specific to the changes we made.

It is probably proliferating though because we are re-generating all your 10 certificates at the same time. However this is the nature of how NixOS's switch-to-configuration script works and I am not sure how to work around this issue. If you had a nixos configuration with 20 certs in 19.09 and you tried to apply it at once it would cause rate-limits too.

To work around these issues, you could perhaps employ SANs. You can add extra SAN domains to a certificate through the
security.acme.certs.<name>.extraDomains option.

[arianvp]

Hmmm Maybe it's actually recreating the account every time... let me debug and see if that's the case...

[arianvp]

Doesn't seem to be recreating the account. So it indeed seems youre running into certificate ratelimits instead of account rate limits. Would you be so kind to provide logs of the failing renewals? I would ike to know exactly why they are failing.

[root@arianvp:/var/lib/acme/.lego]# sha256sum accounts/acme-v02.api.letsencrypt.org/[email protected]/account.json 
b40a96eecd5c6750ea8a3ec59ae8f1b5387057ae3bd84c82cbc01ec0e5a97850  accounts/acme-v02.api.letsencrypt.org/[email protected]/account.json
[root@arianvp:/var/lib/acme/.lego]# sha256sum accounts/acme-v02.api.letsencrypt.org/[email protected]/keys/[email protected] 
4ce788e135006e243ebf6af8c2f40c44fa3dad4b09deda45abdbbdba8e017701  accounts/acme-v02.api.letsencrypt.org/[email protected]/keys/[email protected]


[root@arianvp:/var/lib/acme/.lego]# mv techstock.photos techstock.photos.bak.bak

[root@arianvp:/var/lib/acme/.lego]# systemctl restart acme-techstock.photos.service 

[root@arianvp:/var/lib/acme/.lego]# sha256sum accounts/acme-v02.api.letsencrypt.org/[email protected]/account.json 
b40a96eecd5c6750ea8a3ec59ae8f1b5387057ae3bd84c82cbc01ec0e5a97850  accounts/acme-v02.api.letsencrypt.org/[email protected]/account.json
[root@arianvp:/var/lib/acme/.lego]# sha256sum accounts/acme-v02.api.letsencrypt.org/[email protected]/keys/[email protected] 
4ce788e135006e243ebf6af8c2f40c44fa3dad4b09deda45abdbbdba8e017701  accounts/acme-v02.api.letsencrypt.org/[email protected]/keys/[email protected]

[arianvp]

extraDomains does have the issue at the moment that once you set it; it isn't updated anymore. #84633 (see last bullet point)

The fix for this I will probably try to get backported

[rkoe]

So it indeed seems youre running into certificate ratelimits instead of account rate limits.

No, I'm running into accout rate limits (I've replaced the real domain name with example.com):

2020/04/23 14:24:12 Account [email protected] is not registered. Use 'run' to register a new account.
2020/04/23 14:24:12 [INFO] acme: Registering account for [email protected]
2020/04/23 14:24:13 Could not complete registration
        acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-acct :: urn:ietf:params:acme:error:rateLimited :: Error creating new account :: too many registrations for this IP: see https://letsencrypt.org/docs/rate-limits/, url:

[rkoe]

Thanks for your coments.

I think I've found out:

• I'm using hostmaster@mydomain as acme.certs.<name>.email for the certificates, with mydomain replaced by the actual domain.
• This worked well with simp_le, but lego seems to create a new account for each email.
• Since I'm using 12 different email-adresses, it failed on the last two due to the max-10-accounts-per-3h-and-ip rate limit.

So:

• Will this happen at each certificate renewal, or only once during the simp_le-lego-transition?
• If this only happens during transition: Since I guess that this will not be fixed: At least a note/warning about this should be added to
  • the release notes for 20.03
  • the acme.certs.<name>.email-description

[arianvp]

Aha; yes I see. Thanks for the digging. This makes sense. I think your assesment is completely correct.

@rkoe from what I remember; and correct me if I am wrong, simp_le actually created an account per domain; instead of an account per email. The current situation is hence dramatically better than before as one account can be re-used for multiple domains exactly to resolve rate-limits.

• Will this happen at each certificate renewal, or only once during the simp_le-lego-transition?

Only once. per account

Adding a warning in release notes and to the description

This sounds like a good idea. Especially the remark that each email creates a new account... and that grouping multiple domains under the same email can help with this issue.

[m1cr0man]

@arianvp I know you edited your message but I just wanted to confirm that yeah, it used to be one account per certificate, or in other words it was 1:1 between security.acme.certs.<name> and accounts. Now it is 1:1 with email addresses as observed above.

It's also worth pointing out that the security.acme.server is taken into consideration by lego too, so changing that will cause new accounts to be created too. This comes into play if you are swapping between staging and prod LE servers.

[arianvp]

So I think the conclusion is we're in a better shape than before. But adding a warning about rate limits especially if you have more than 10 accounts could be added to the release notes so people do not run into this.

I'll make a PR. @rkoe sounds good?

[rkoe]

Yes, thanks.

Additionally, a warning at the acme.certs.<name>.email-description [1] would be quite useful, since this would help users not to trigger the rate limit.

[1] e.g.: For each distinctive email-address, a separate account is registered at the ACME-server (e.g. let's encrypt). Note that ACME-servers usually have account rate limits (e.g. max. 10 per IP per 3 hours), so adding several acme-entries with different email-addresses may trigger those limits.

[stale]

Hello, I'm a bot and I thank you in the name of the community for opening this issue.

To help our human contributors focus on the most-relevant reports, I check up on old issues to see if they're still relevant. This issue has had no activity for 180 days, and so I marked it as stale, but you can rest assured it will never be closed by a non-human.

The community would appreciate your effort in checking if the issue is still valid. If it isn't, please close it.

If the issue persists, and you'd like to remove the stale label, you simply need to leave a comment. Your comment can be as simple as "still important to me". If you'd like it to get more attention, you can ask for help by searching for maintainers and people that previously touched related code and @ mention them in a comment. You can use Git blame or GitHub's web interface on the relevant files to find them.

Lastly, you can always ask for help at our Discourse Forum or at #nixos' IRC channel.

[m1cr0man]

This issue was resolved in #91121 and I did add a note about using different emails for lots of certs too.