Migrating a Production Server

The following steps outline the procedure for migrating a production server.

Setting up the new server

Get a new server. Create a temporary subdomain for it, eg. migration.openfoodnetwork.com. The DNS will take a while to propagate, but you can start with just the new IP address.

Add a new entry in your copy of ofn-install in the /inventory/hosts file, and a new directory in /inventory/host_vars/ for the new server.

In the host_vars entry for the new server (migration.openfoodnetwork.com), define the certbot_domains and certbot_cert_name variables, and set them like this:

certbot_domains:
  - openfoodnetwork.com           # Current domain FIRST in the list
  - www.openfoodnetwork.com       # www subdomain, if you use it
  - migration.openfoodnetwork.com # New domain LAST

certbot_cert_name: openfoodnetwork.com # Current domain name

In the hosts file, add the new entry alongside the current entry. It helps to make sure both IP addresses are defined with ansible_host, eg:

[au-prod]
openfoodnetwork.org.au ansible_host=43.239.97.238

Provisioning the new server ⚠️

Before provisioning, run the letsencrypt_proxy.yml playbook on the current production server, and specify proxy_target with the domain (without the http:// prefix) or the IP address of the new server, eg:

ansible-playbook playbooks/letsencrypt_proxy.yml --limit au-prod -e "proxy_target=migration.openfoodnetwork.org.au"
OR 
ansible-playbook playbooks/letsencrypt_proxy.yml --limit au-prod -e "proxy_target=123.456.78.9"

The above hack (and the earlier config steps) should ensure the new server can successfully create a Letsencrypt certificate during provisioning that will be valid for any of the listed subdomains.

When that's done, follow the normal steps for setup, provision, and deploy on the new server. Make sure you deploy the same release as the current server!

Migrating custom assets

The current server may have custom assets such as instance-specific logos, or PDFs with the instance's Terms and Conditions in a localised language. These are generally located under: /home/openfoodnetwork/apps/openfoodnetwork/shared/.

Use the assets_pull.yml playbook on the current server, then assets_push.yml on the new server. The assets will show up on the new site when it's deployed again.

The switchover ⚠️

Announce some downtime to users, and get ready for the switch. At the scheduled time, follow these steps:

• Put the current server into maintenance mode with the maintenance_mode playbook. The default message is in English, but you can specify a custom message like so: -e "maintenance_mode_message='Estamos manteniendo el servidor'
• Run the db_backup playbook again on the current server, then run the db_restore playbook again on the new server. Take a look at the db_transfer.yml playbook for a faster way to dump, transfer, and then restore the database on the new server 🚀.
• Run the temporary_proxy playbook on the current server, specifying the IP address of the new server with the extra_vars option, eg: -e 'proxy_target=123.456.7.8'.
• You might need to do a unicorn restart at this point to pick up the new data, like mail configs and site settings.

This should only take a couple of minutes, and the site will now be back up, with requests to the old server being transparently proxied to the new one.

Finishing up

If it's all working, you can change the DNS of the domain name from the old server's IP to the new server's IP. It can theoretically take up to 48 hours to propagate, but the change will appear instant to users because of the temporary_proxy.yml playbook.

You can also run the playbooks for asset migration again (push and pull) to pick up any new changes, if the images are hosted on the server. The tasks in the playbook are idempotent and should cleanly update only the changed or missing assets.

After 48 hours you can shut down the old server, and make sure the entries in ofn-install are up to date: remove the temporary entry made for the migration, and set the new IP address.