Migrating a Production Server

The following guide outlines the procedure for migrating an OFN instance to a new server. For a step-by-step checklist see the issue template: https://github.com/openfoodfoundation/ofn-install/issues/new?template=server-migration.md

Note that it is more efficient to migrate without provisioning with the temporary subdomain, but a process for that hasn't been recorded.

Setting up the new server

Check all the requirements for server (for example system specs, and are any other services running on it?)

Get a new server. Create a uniqe subdomain for it, eg. prod2.openfoodnetwork.com. The DNS will take a while to propagate, but you can start with just the new IP address.

Add a new entry in your copy of ofn-install in the /inventory/hosts file, and a new directory in /inventory/host_vars/ for the new server.

In the host_vars entry for the new server (prod2.openfoodnetwork.com), define the domain variables, and set them like this:

domain: prod2.openfoodnetwork.ca # might be required for testing. remember to change before switchover
host_id: ca-prod2
...

certbot_cert_name: openfoodnetwork.com # Current domain name

certbot_domains:
  - openfoodnetwork.com           # Current domain FIRST in the list
  - www.openfoodnetwork.com       # www subdomain, if you use it
  - prod2.openfoodnetwork.com     # New domain LAST

In the hosts file, add the new entry alongside the current entry. It helps to make sure both IP addresses are defined with ansible_host, eg:

[x_prod]
openfoodnetwork.com ansible_host=123.183.204.127
[x_prod2]
prod2.openfoodnetwork.com ansible_host=123.190.252.221

Provisioning the new server ⚠️

Before provisioning, run the letsencrypt_proxy.yml playbook on the current production server, and specify proxy_target with the domain or the IP address of the new server, eg:

ansible-playbook playbooks/letsencrypt_proxy.yml -l x_prod -e "proxy_target=prod2.openfoodnetwork.com"

The above hack (and the earlier config steps) should ensure the new server can successfully create a Letsencrypt certificate during provisioning that will be valid for any of the listed subdomains.

When that's done, and the DNS has propagated for the temporary domain, follow the normal steps for setup, provision, and deploy on the new server. You can refer to the secrets for the existing server, eg:

ansible-playbook site.yml -l x_prod2 -e "@../ofn-secrets/x_prod/secrets.yml"

Important

Subscriptions may start getting processed. Stop and disable the sidekiq service to avoid this.

Migration

SSH forwarding speedup

If you have a lot of data or need to minimise downtime, it's worth setting up an SSH key to allow direct transfer from the old to new server. In short:

For db_transfer: ofn-admin@x-prod2: ssh-keygen -t ed25519 && cat ~/.ssh/id_ed25519.pub ofn-admin@x-prod: vim .ssh/authorized_keys

For transfer_assets: openfoodnetwork@x-prod2: ssh-keygen -t ed25519 && cat ~/.ssh/id_ed25519.pub openfoodnetwork@x-prod: vim .ssh/authorized_keys

You might need to manually ssh first to ensure it's added to known_hosts

Custom assets

The current server may have custom assets such as instance-specific logos, or PDFs with the instance's Terms and Conditions in a localised language. These are generally located under: /home/openfoodnetwork/apps/openfoodnetwork/shared/.

If local storage is used instead of S3, a lot of images will be stored under storage. You will want to copy assets directly to the new server with transfer_assets.yml, making sure to set up authorised key:

ansible-playbook playbooks/transfer_assets.yml -l x_prod -e "rsync_to=x_prod2"

If you haven't got the authorised key setup, just use the pull_assets.yml playbook on the current server, then push_assets.yml on the new server. The assets will show up on the new site when it's deployed again.

ansible-playbook playbooks/pull_assets.yml -l x_prod
ansible-playbook playbooks/push_assets.yml -l x_prod2

Sidekiq job data

It's important to copy the sidekiq redis DB. But keep sidekiq offline for now, to avoid subscriptions being taken etc.

# old server: this file will be dumped when redis-jobs is stopped.
ofn-admin@ca-prod:~$ sudo systemctl stop redis-jobs
ofn-admin@ca-prod:~$ sudo cp /var/lib/redis-jobs/dump.rdb ~/
ofn-admin@ca-prod:~$ sudo chown ofn-admin: ~/dump.rdb
# new server: 
ofn-admin@ca-prod2:~$ sudo systemctl stop redis-jobs
ofn-admin@ca-prod2:~$ rsync openfoodnetwork.com:dump.rdb ~/
ofn-admin@ca-prod2:~$ sudo cp ~/dump.rdb /var/lib/redis-jobs/
ofn-admin@ca-prod2:~$ sudo systemctl start redis-jobs

When redis starts, it will automatically load this file.

Deploy

Ensure sure it has the exact same release as the current server:

ssh [email protected] -C "(cd ~/apps/openfoodnetwork/current && git show)"
# (refer to commit on latest release: https://github.com/openfoodfoundation/openfoodnetwork/releases)
ansible-playbook playbooks/deploy.yml -l x_prod2 -e "git_version=vX.Y.Z"

Testing

If the new server is a subdomain of the old one, you may end up with conflicting cookies. In some cases, it will be necessary to open devtools and manually remove the cookies for old prod, just before clicking "login". After logging in, it should work.

Test emails at Admin > Configuration > Mail Methods > Send test email.

The switchover ⚠️

Announce some downtime to users, and get ready for the switch. At the scheduled time, follow these steps:

1. Put the current server into maintenance mode with the maintenance_mode playbook. The default message is in English, but you can specify a custom message like so: -e "maintenance_mode_message='Estamos manteniendo el servidor'

   #ansible-playbook playbooks/maintenance_mode.yml -l x_prod
   

2. Stop sidekiq and redis-jobs

sudo systemctl stop sidekiq redis-jobs 

2. Transfer database and files:

ansible-playbook playbooks/db_transfer.yml -l x_prod -e rsync_to=x_prod2 && ansible-playbook ansible-playbook playbooks/transfer_assets.yml -l x_prod -e "rsync_to=x_prod2"

4. Transfer redis-jobs db^
5. sudo systemctl stop puma postgres
6. restart puma and start sidekiq and redis-jobs on the new server at this point to pick up the newly imported data, like mail configs and site settings.
7. Run the temporary_proxy playbook on the current server, specifying the IP address of the new server with the extra_vars option, eg: -e 'proxy_target=123.456.7.8'.

This should not take too long, and after the last step the site will now be back up, with requests to the old server being transparently proxied to the new one.

Finishing up

If it's all working, you can change the DNS of the domain name from the old server's IP to the new server's IP. It can theoretically take up to 48 hours to propagate, but the change will appear instant to users because of the temporary_proxy.yml playbook.

You can also run the playbooks for asset migration again (push and pull) to pick up any new changes, if the images are hosted on the server. The tasks in the playbook are idempotent and should cleanly update only the changed or missing assets.

After 48 hours you can shut down the old server, and make sure the entries in ofn-install are up to date: remove the temporary entry made for the migration, and set the new IP address.