Migrating a Production Server

The following steps outline the procedure for migrating a production server.

Setting up the new server

Get a new server. Create a temporary subdomain for it, eg. migration.openfoodnetwork.com. The DNS will take a while to propagate, but you can start with just the new IP address.

Add a new entry in your copy of ofn-install in the /inventory/hosts file, and a new directory in /inventory/host_vars/ for the new server.

In the host_vars entry for the new server (migration.openfoodnetwork.com), define the certbot_domains and certbot_cert_name variables, and set them like this:

certbot_domains:
  - openfoodnetwork.com           # Current domain FIRST in the list
  - www.openfoodnetwork.com       # www subdomain, if you use it
  - migration.openfoodnetwork.com # New domain LAST

certbot_cert_name: openfoodnetwork.com # Current domain name

In the hosts file, add the new entry and make sure both IP addresses are defined. You should also define the ofn_80 variable in the host_vars of the current server and set it to proxy the Letsencrypt check to the new server's domain (or IP address). Examples for the above are here: https://github.com/openfoodfoundation/ofn-install/pull/454/files

When that's done, follow the normal steps for setup, provisioning, and deployment of the new server. Make sure you deploy the same release as the current server.

Migrating custom assets

The current server may have custom assets such as instance-specific logos, or PDFs with the instance's Terms and Conditions in a localised language. These are generally located under: /home/openfoodnetwork/apps/openfoodnetwork/shared/.

Use the assets_pull.yml playbook on the current server, then assets_push.yml on the new server. They won't show up on the new site until the next step:

Migrating the data

To check if everything is ok (as a sort of dry-run) you can do an initial data migration whilst both servers are live. Use the db_backup.yml playbook on the current server, then use the db_restore.yml playbook on the new server. If there's any issues, you may need to restart nginx and/or unicorn or deploy again to the the new server.

The new server should now be an exact copy of the current one (but not in sync). Take some time to check everything is working and nothing is missing. I everything looks ok, you can proceed to the next steps.

The switchover ⚠️

Announce some downtime to users, and get ready for the switch. At the scheduled time, follow these steps:

• Put the current server into maintenance mode with the maintenance_mode playbook. The default message is in English, but you can specify a custom message like so: -e "maintenance_mode_message='Estamos manteniendo el servidor'
• Run the db_backup playbook again on the current server
• Run the db_restore playbook again on the new server
• Run the temporary_proxy playbook on the current server, specifying the IP address of the new server with the extra_vars option, eg: -e 'proxy_target=192.168.0.1'.

This should only take a couple of minutes, and the site will now be back up, with requests from the old server being transparently proxied to the new one.

Finishing up

If it's all working, you can change the DNS of the domain name from the old server's IP to the new server's IP. It'll take up to 48 hours to actually propagate, but the change will appear instant to users as we are using temporary_proxy.yml.

You can also run the playbooks for asset migration again (push and pull) to pick up any changes, if the images are hosted on the server. The tasks in the playbook are idempotent and should cleanly update only the changed or missing assets.

After 48 hours you can shut down the old server, and make sure the entries in ofn-install are up to date: remove the temporary entry made for the migration, and set the new IP address.