Restrict SSH to all VMs from VPN

[dduportal]

Following CVE-2024-6387 (OpenSSH), we can improve the safety of the Jenkins Infra by restricting the inbound SSH connections on our VMs to only restricted subsets.

It would allow limiting the brute force attacks and other issues like this CVE. Of course it is not an absolute protection but it adds another layer of safety.

We have 4 kinds of VMs with different restrictions to apply (or already there):

• AWS (CloudBees) account VMs:

  • There are 3 VMs:
    • pkg (hosting pkg.origin.jenkins.io and updates.jenkins.io)
    • census (hosting census.jenkins.io)
    • usage (hosting usage.jenkins.io)
  • There are no SSH restriction today. Opened a draft PR at feat: restrict SSH access to VMs aws#556 to propose restrictions and only allow inbound SSH from our private networks (trusted.ci for update center2 and release.ci for Core releases).
    • As per @timja proposal, we should restrict to our private networks and VPN, while adding routing through VPN. Would be easier than maintaining outbound IP (when admin travels) => makes sense for these machines which provides, anyway, public services on HTTP/HTTPS (e.g. public facing) as the risk of the VPN being pwned does not balance the PITA to maintain public IPs.
  • 🎯 Restrict access to these 3 VMs through VPN (or trusted or private vnets)

• DigitalOcean VMs:

  • There is only 1 VM: archives (hosting archives.jenkins.io)
  • There are no SSH restriction today.
  • 🎯 Same as AWS: Restrict access to this VMs through VPN (or trusted or private vnets)

• Azure private VMs:

  • All these VMs are already restricted for SSH as they only are reachable by the private network (and require VPN)

• Azure public VMs:

  • ci.jenkins.io is restricted to get SSH through a secondary network interface which is private (while public network interface allows inbound HTTP/S).
  • puppet and VPN VMs are restricted by admin IPs and it is OK like this

[timja]

I would say puppet and VPN VMs should still be accessible via VPN.

Should be possible to access them publicly when VPN is broken but by default shouldn't the same principals apply as the other VMs?

[dduportal]

Should be possible to access them publicly when VPN is broken but by default shouldn't the same principals apply as the other VMs?

• For the VPN VM, it is clearly not possible as it would create a lot of trouble with routing (routing to itself).
• For both Puppet and VPN, what would be the need to access them for non admin with VPN access? Given these 2 VMs are highly sensitive, the additional restriction here (to protect against someone with VPN access, not admin, trying to sniff them) is welcome to be honest
  • If you are admin and need access, you can always add your public IP in the Azure UI or in the Terraform data

[dduportal]

Update: VPN need a bit of work:

• Routing from the VPN to the remote VMs is needed on the VM network setup (iptables NAT rules)
• VPN Docker Image need to have its CCDs updated to get the external IP to route through the VPN connection

=> tested manually with success, incoming PRs to persist and generalize

[dduportal]

Update: VPN routes are ready! Let's go with the 3 AWS machines

[dduportal]

• Restrictions applied everywhere and working with VPN
• No errors on mirrorbrain sync (pkg -> archives) and on update_center2