Upgrade Nginx Plus Load Balancers to Ubuntu Jammy

[kayiwa]

This ticket outlines the plan to upgrade our two Nginx Plus load balancers (lib-adc1 and lib-adc2) from Ubuntu Focal (20.04) to Ubuntu Jammy (22.04) with zero downtime.

Background:

Ubuntu Focal is nearing its end of life. Upgrading to Jammy will provide security updates and new features.
Nginx Plus is critical for our infrastructure, so the upgrade must be performed with zero downtime.
Goals:

Upgrade both lib-adc1 and lib-adc2 to Ubuntu Jammy.
Maintain high availability throughout the upgrade process.
Minimize any disruption to services.

Runbook details

We will use a blue-green deployment strategy to ensure zero downtime:

Provision new servers: Spin up two new servers (adc-prod1 and adc-prod2) with Ubuntu Jammy and the latest version of Nginx Plus.
Configure new load balancers: Replicate the configuration from lib-adc1 and lib-adc2 to adc-prod1 and adc-prod2, ensuring all settings, certificates, and configurations are identical.
Test new load balancers: Thoroughly test adc-prod1 and adc-prod2 to ensure they function correctly and handle traffic as expected.
Redirect traffic to new load balancers: Gradually redirect traffic from lib-adc1 and lib-adc2 to adc-prod1 and adc-prod2. This will be done using DNS changes, VRRP and keepalived in front of the load balancers.

Monitor: Closely monitor the new load balancers for any issues during the traffic transition.
Decommission old load balancers:

Once traffic is fully migrated and the new load balancers are stable, decommission lib-adc1 and lib-adc2.
Rollback Plan:

In case of any issues during the upgrade, we will immediately redirect traffic back to the original load balancers (lib-adc1 and lib-adc2).

Tasks:

• Create new server instances (adc-prod1 and adc-prod2) with Ubuntu Jammy.
• Install and configure Nginx Plus on adc-prod1 and adc-prod2.
• Replicate configuration from lib-adc1/lib-adc2 to adc-prod1/adc-prod2.
• Test adc-prod1 and adc-prod2
• Decommission lib-adc1 and lib-adc2.
• Revoke any unneeded SSL certificates (we currently have 8)

[kayiwa]

I think we can safely decommission the old hardware now.

[aruiz1789]

Working on decommissioning the servers:

• lib-adc1.princeton.edu 128.112.203.145
• lib-adc2.princeton.edu 128.112.203.144

Grant access to both Out of Band VMs to allow iDrac interaction.

• Forrestal: lib-servoobm01.princeton.edu:9833
• New South - lib-servoobm02.princeton.edu:9833

[aruiz1789]

Both servers have been powered down and the disks will be formated in two weeks.