Valid ssl certificate for trusted.ci.jenkins.io

[timja]

Service(s)

trusted.ci.jenkins.io

Summary

Relates to #2059

For the second time something has set hsts for the jenkins.io domain preventing me from accessing trusted.ci without clearing the flag on the domain

HSTS settings

Found:
static_sts_domain:
static_upgrade_mode: UNKNOWN
static_sts_include_subdomains:
static_sts_observed:
static_pkp_domain:
static_pkp_include_subdomains:
static_pkp_observed:
static_spki_hashes:
dynamic_sts_domain: jenkins.io
dynamic_upgrade_mode: FORCE_HTTPS
dynamic_sts_include_subdomains: true
dynamic_sts_observed: 1659962059.92962
dynamic_sts_expiry: 1660048459.929614
static_sts_domain:
static_upgrade_mode: UNKNOWN
static_sts_include_subdomains:
static_sts_observed:
static_pkp_domain:
static_pkp_include_subdomains:
static_pkp_observed:
static_spki_hashes:
dynamic_sts_domain: jenkins.io
dynamic_upgrade_mode: FORCE_HTTPS
dynamic_sts_include_subdomains: true
dynamic_sts_observed: 1659962059.92962
dynamic_sts_expiry: 1660048459.929614
static_sts_domain:
static_upgrade_mode: UNKNOWN
static_sts_include_subdomains:
static_sts_observed:
static_pkp_domain:
static_pkp_include_subdomains:
static_pkp_observed:
static_spki_hashes:
dynamic_sts_domain: jenkins.io
dynamic_upgrade_mode: FORCE_HTTPS
dynamic_sts_include_subdomains: true
dynamic_sts_observed: 1659962059.92962
dynamic_sts_expiry: 1660048459.929614

It would be good to have a valid cert on there, (I wasn't able to find an existing issue for this but there may be one)

Reproduction steps

No response

[dduportal]

That make sense! @daniel-beck asked the same a few months ago, but the infra team never took the time.
Having the issue will help.

Givent that this machine is not directly reachable from the internet, the proposal is to use a Letsencrypt DNS validation, either with:

• A dummy ingress in Kubernetes (that already has cert-manager set up with DNS validation for jenkins.io and jenkins-cin.org domains) and then we retrieve the certificate to the VM hosting trusted.ci.jenkins.io
• Use the certbot DNS modules (ref. https://www.digitalocean.com/community/tutorials/how-to-acquire-a-let-s-encrypt-certificate-using-dns-validation-with-acme-dns-certbot-on-ubuntu-18-04) on the VM itself to generate/install the certificate. That would require a manual step (to create and delete the DNS TXT record) on each renewal. Not a problem as it's easy, but not fully automated.
• Add Traefik in front of trusted.ci.jenkins.io, with the Azure DNS credentials, to fulfill an entire automated renewal (unlike certbot)

Long term: migrating trusted.ci.jenkins in the private Kubernetes (once #2844 is done) will solve this easily (like release.ci).

[timja]

• Use the certbot DNS modules

Can't this be used to automate?
https://certbot-dns-azure.readthedocs.io/en/latest/

[dduportal]

• Use the certbot DNS modules

Can't this be used to automate? https://certbot-dns-azure.readthedocs.io/en/latest/

Oooh that would be the easier way for this particular machine (private machine only reachable through tunnels by a specific set of persons) so the DNS credentials are ok to be stored on the same machine (under a different account than jenkins of course).

We can also do the same for cert.ci.jenkins.io (cc for info. @Wadeck @daniel-beck ).

Please note that, technically we could do the same to ci.jenkins.io but we won't given the nature of this service.

[timja]

there's 2 options for restricting further access to the DNS records:

1. Create a separate zone and grant access to the SP for that zone
2. Assign DNS Contributor to the individual records and not the whole zone

[dduportal]

there's 2 options for restricting further access to the DNS records:

1. Create a separate zone and grant access to the SP for that zone

2. Assign DNS Contributor to the individual records and not the whole zone

I was thinking implictly on the second (e.g. a technical user restricted to only the trusted.ci.jenkins.io domain in this case).

[timja]

generally there's a sub record for the validation something like _dnschallenge.trusted.ci.jenkins.io

[dduportal]

cc @lemeurherve as this service runs in AWS: the Route53 DNS delegation could help for this

[dduportal]

Worklog after #3328 and the pairing session with @lemeurherve:

• The puppet letsencrypt module that we are using does not support DNS-Azure out of the box: we'll need to setup (with puppet) certbot:

  • We are using the old 6.0.0 version of the module
  • The latest (9.0.1) of the module does not support dns-azure either
  • There is an open PR (Add dns-azure to allowed plugins voxpupuli/puppet-letsencrypt#298) for it, quite recent. 🤞 The PR indicates a workaround (see below)
  • Current target (workaround): installing certbot and certbot-dns-azure plugin with snap (trusted.ci runs on ubuntu 18.04 as for today)

• Azure API protection to allow DNS challenge with a certain level of safety: As mentioned by @timja we'll create a DNS (child) zone for trusted.ci and secure it with a Service Principal (the VM runs on AWS EC2 so no workload identity. Long term: we should migrate trusted.ci to Azure).

[dduportal]

First challenge: the snap install of certbot-dns-azure on Ubuntu 18.04 fails when executing certbot with the error

certbot plugins --text
An unexpected error occurred:
pkg_resources.VersionConflict: (certbot 2.2.0 (/snap/certbot/2683/lib/python3.8/site-packages), Requirement.parse('certbot<2.0,>=1.18.0'))
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/certbot-log-xl7x0e7w/log or re-run Certbot with -v for more details.

I'm currently playing around with a Dockerfile found in terricain/certbot-dns-azure#22 (considered experimental for now) and was able to succesfully execute azure DNS integration with the certbot in the container:

docker run --rm certbot certbot plugins --text
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* dns-azure
Description: Obtain certificates using a DNS TXT record (if you are using Azure
for DNS).
Interfaces: Authenticator, Plugin
Entry point: dns-azure = certbot_dns_azure._internal.dns_azure:Authenticator

* standalone
Description: Spin up a temporary webserver
Interfaces: Authenticator, Plugin
Entry point: standalone = certbot._internal.plugins.standalone:Authenticator

* webroot
Description: Place files in webroot directory
Interfaces: Authenticator, Plugin
Entry point: webroot = certbot._internal.plugins.webroot:Authenticator
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

=> I'm defering the "certbot install for later and continuing to work on the DNS-01 challenge from the trusted.ci.jenkins.io machine.

[dduportal]

Second step: after manually (e.g. locally) creating (with Terraform) the Azure resources for the child zone (+SP + role association scoped to the child zone only), I was able to successfully generate a staging LE certificate.

A few notes:

• A child zone of an existing DNS zone requires an NS record in the parent zone (ref. Support for creating child DNS-zone hashicorp/terraform-provider-azurerm#14733 (comment))
• The azurerm_dns_zone.id attribute is enough to pass to the role assignement scope \o/
• The certbot certonly command to execute require a file with the credentials (to be added through the eyaml encrypted data) with a 0600 mode (otherwise it fails) + a specified DNS propagation of 30 s (--dns-azure-propagation-seconds 30)

Incoming PRs...

[root-expert]

First challenge: the snap install of certbot-dns-azure on Ubuntu 18.04 fails when executing certbot with the error

certbot plugins --text
An unexpected error occurred:
pkg_resources.VersionConflict: (certbot 2.2.0 (/snap/certbot/2683/lib/python3.8/site-packages), Requirement.parse('certbot<2.0,>=1.18.0'))
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/certbot-log-xl7x0e7w/log or re-run Certbot with -v for more details.

You can track certbot's latest/stable channel (version 1.32.2 as of now) and it should work, except if you have specific requirements for latest/beta (version 2.2.0)

[dduportal]

(edit after a lot of changes in Terraform)

• feat(dns) add letsencrypt DNS support for trusted.ci.jenkins.io  azure-net#30 => Initial PR to manage DNS zones and associated Service Principal
• Since the deployment failed, we tried to cheat and use a single Azure AD App in the (private) repo jenkins-infra/terraform-states and reference it through a data source in jenkins-infra/azure-net
  • (private) https://github.com/jenkins-infra/terraform-states/commit/a671b2063bfbe41055dbee5fffb69430ed4d9827
  • jenkins-infra/azure-net@77267a9
• Of course it also failed: we had to struggle (a bit ?) with Azure AD permissions as pointed out by @timja in feat(dns) add letsencrypt DNS support for trusted.ci.jenkins.io  azure-net#30 (comment).
  • Reverting back to "1 app per Let's Encrypt app in azure-net": Revert "hotfix(dns) use existing Azure AD application for Let's Encrypt instead of managing it(s)" azure-net#32 (+cleanup in the -private- repo - https://github.com/jenkins-infra/terraform-states/commit/841ac4f9d375ed907d6844e88dae7875311c025d) => still failure (due to missing permissions for the Terraform Service Principal)
  • We switched from "SP password" to "Azure AD App password" to avoid granting full admin to AD for terraform: as Application.ReadWrite.OwnedBy is only supported by the former: fix(dns) use ad_application_password instead of sp_password azure-net#33
  • Missing permission to allow the Terraform SP to assign/unassign roles: https://github.com/jenkins-infra/terraform-states/commit/498908dc979d69f216e617b5d1256fa139b67bfe

(We do not details ALL the time spent and the die & retries of course...)

(https://twitter.com/fredkisss/status/1614384130908643330?s=20 )

[dduportal]

Next step: jenkins-infra/jenkins-infra#2587 (to avoid dealing with certbot): the puppet module for LE is now supporting DNS azure \o/

[timja]

Why create a custom role instead of using User Access Administrator?

[dduportal]

Why create a custom role instead of using User Access Administrator?

As I understand https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles#user-access-administrator, the permissions associated to the built-in role are far too extended for the terraform-azure* Service principal used by terraform (support tickets, all authorizations)

[timja]

That built-in role is designed for delegation of access.

The two permissions on it as far as I understand are creating of roles and assigning of roles.

(Support tickets is standard on the majority of built-in roles)

For larger scale places at least I would also suggest better names and descriptions for the role.
terraform_role_allowRoleAssign will show up in the UI and look awful compare to the existing naming there.

User role assignment maybe and the description should say what it does

[dduportal]

That built-in role is designed for delegation of access.

Yes, which is not what we want for the terraform SPs. The docs states that it gives all of https://learn.microsoft.com/en-us/azure/role-based-access-control/resource-provider-operations#microsoftauthorization
=> we definitively do not want to give these accesses to these SP (case of infra.ci compromised, or these SP tokens exposed).

Good thing, thanks to your pointers and questions I know start to better understand the permissions model \o/

For larger scale places at least I would also suggest better names and descriptions for the role.
terraform_role_allowRoleAssign will show up in the UI and look awful compare to the existing naming there.

User role assignment maybe and the description should say what it does

👍 make sense, gotta update it (I'll ask you for review in the terraform PR). Thanks for all the tips!

[dduportal]

Done in

• feat(trusted.ci.jenkins.io) use Let's Encrypt DNS challenge with certbot-dns-azure plugin jenkins-infra#2607
• fixup in Hotfix/helpdesk 3091 jenkins-infra#2611
• tests fixup in chore(rspec) fix jenkins role unit tests - fixup of 3ac9b8b9 jenkins-infra#2612