Migrating a Production Server

The following steps outline the procedure for migrating a production server.

Setting up the new server

Get a new server. Create a temporary subdomain for it, eg. migration.openfoodnetwork.com. The DNS will take a while to propagate, but you can start with just the new IP address.

Add a new entry in your copy of ofn-install in the /inventory/hosts file, and a new directory in /inventory/host_vars/ for the new server.

In the host_vars entry for the new server (migration.openfoodnetwork.com), define the certbot_domains and certbot_cert_name variables, and set them like this:

certbot_domains:
  - openfoodnetwork.com           # Current domain FIRST in the list
  - www.openfoodnetwork.com       # www subdomain, if you use it
  - migration.openfoodnetwork.com # New domain LAST

certbot_cert_name: openfoodnetwork.com # Current domain name

In the hosts file, add the new entry and make sure both IP addresses are defined. You should also define the ofn_80 variable in the host_vars of the current server and set it to proxy the Letsencrypt check to the new server's domain (or IP address). Examples for the above are here: https://github.com/openfoodfoundation/ofn-install/pull/454/files

When that's done, follow the normal steps for setup, provisioning, and deployment of the new server. The above hack should ensure the new server can successfully create a Letsencrypt certificate during provisioning that will be valid for any of the domain names. You can remove the proxying after it's done.

Make sure you deploy the same release as the current server.

Migrating custom assets

The current server may have custom assets such as instance-specific logos, or PDFs with the instance's Terms and Conditions in a localised language. These are generally located under: /home/openfoodnetwork/apps/openfoodnetwork/shared/.

Use the assets_pull.yml playbook on the current server, then assets_push.yml on the new server. The assets will show up on the new site when it's deployed again.

The switchover ⚠️

Announce some downtime to users, and get ready for the switch. At the scheduled time, follow these steps:

• Put the current server into maintenance mode with the maintenance_mode playbook. The default message is in English, but you can specify a custom message like so: -e "maintenance_mode_message='Estamos manteniendo el servidor'
• Run the db_backup playbook again on the current server
• Run the db_restore playbook again on the new server
• Run the temporary_proxy playbook on the current server, specifying the IP address of the new server with the extra_vars option, eg: -e 'proxy_target=123.456.7.8'.
• You might need to do a unicorn restart at this point to pick up the new data, like mail configs and site settings.

This should only take a couple of minutes, and the site will now be back up, with requests to the old server being transparently proxied to the new one.

Finishing up

If it's all working, you can change the DNS of the domain name from the old server's IP to the new server's IP. It'll take up to 48 hours to actually propagate, but the change will appear instant to users as we are using temporary_proxy.yml.

You can also run the playbooks for asset migration again (push and pull) to pick up any changes, if the images are hosted on the server. The tasks in the playbook are idempotent and should cleanly update only the changed or missing assets.

After 48 hours you can shut down the old server, and make sure the entries in ofn-install are up to date: remove the temporary entry made for the migration, and set the new IP address.