Updating

Steps Overview (TOC)

1. Preparation
2. Backup
3. New Deployment

Preparation

As early as possible

Preferably, this should be done on the previous day(s). Tasks marked with a tick may be skipped safely.

• Give a heads-up to both Bioinformatics and Sequencing Facility that you're about to do this.
• Check that the VM has enough storage, df -h, it should be able to accommodate two copies of your media_dump (e.g. 10 GiB).
• If needed, update the User manual accordingly.
• If there were changes to models. On parkour-dev, make update-fixtures. Then, stage and commit anything needed.
  • Formatting will be done by pre-commit hook. (Differences like the JSON values that are lists spanning multiple lines can be staged to commit.)
  • WARNING: librarypreparation.json had some fields reset to null the last time. Was that a bug? I deliberately didn't commit that, only the new field that was added to our model was reflected in the JSON.
• If there was a new TLS certificate issued, replace misc/cert.pem with it.
  • Permissions on these files were never an issue, but it's an insidious error if it ever happens. Furthermore, in our case, both key and chain usually stay the same. If that's not the case, check: misc/key.pem and cat 🐱

Before moving forward

We need to make a new release here, with a new version number assigned. So...

• Look at Actions, everything should be passing.
• Proofread updated CHANGELOG.md file contents, it should reflect the most relevant PRs.
• Adjust release version with current date/ version number at backend/common/templates/login.html, add, commit, and push those changes to the remote repo.
• git tag yy.mm.dd && git push --tags
• Make a pre-release on GitHub, with date/ version number as title and changelog contents as description.

---

Note

We have two VMs. First, production (hostname: parkour), and secondly, a staging environment (hostname: parkour-dev). In both cases, the directory that holds the repo is ~/parkour2 .

Backup

In order to minimize downtime, we can build the new images while going through the backup process. So, ssh into production VM and run:

# build the new images
mkdir -p ~/upgrade
cd ~/upgrade                                                                                            
git clone --depth=1 https://github.com/maxplanck-ie/parkour2
cd parkour2
docker compose build

Leave that running. On another shell, ssh into production again, and then:

make --directory ~/parkour2 sweep
rm -rf ~/pk2_old
date >> ~/last.txt
docker ps >> ~/last.txt
git -C ~/parkour2 log | head -1 >> ~/last.txt
make --directory ~/parkour2 save-media

Meanwhile, go into your development/ staging VM (e.g. parkour-dev) and start the app that's a mirror (code and data) of production:

ssh root@parkour-dev
cd parkour2
make down
git pull
git checkout $(git tag | tail -2 | head -1)  # previous release
make import-pgdb dev db-migras
docker exec parkour2-django python manage.py check

Open this app in a browser tab, navigate submodules (Library Preparation, Index Generator, Pooling, Load Flowcells, etc.); and confirm that all numbers look the same as production. We'll come back to this mirror to check everything is good, once we finish the whole process...

Warning

Only proceed if the save-media process that was left in a previous terminal has actually finished.

Now, take some last backup measures, and turn everything off:

docker exec parkour2-rsnapshot rsnapshot halfy
make --directory ~/parkour2 save-postgres tar-old-migras
cd ~/parkour2; docker-compose -f docker-compose.yml -f nginx.yml -f rsnapshot.yml stop; cd -
make --directory ~/parkour2 clean
mv ~/parkour2 ~/pk2_old

The ~/pk2_old folder is for quick and easy restoration, you will simply need to come back to this folder, rename it, and run the corresponding make commands to re-deploy the service with the proper backup data that's in place.

New Deployment

Warning

Only move to this section after the docker compose build has finished. Ideally, this was run in previous days and a new recent re-build would only take 2-3 minutes.

Install

mv ~/upgrade/parkour2 ~/
ln -s /parkour/backups ~/parkour2/misc/rsnapshot/backups

Note

Our rsnapshot backups goes to the ZFS Volume mounted at /parkour/backups, for that matter we're using such a symbolic link ;)

Config & Go LIVE!

cp ~/pk2_old/misc/parkour.env.ignore ~/parkour2/misc/
cp ~/pk2_old/misc/migras.tar.gz ~/parkour2/misc/
cp -av ~/pk2_old/misc/*.sqldump ~/parkour2/misc/
cp ~/pk2_old/misc/*.pem ~/parkour2/misc/
sync
make --directory ~/parkour2 prod clearpy prod db-migras

In this command, we are now assuming there were changes to any models' field, otherwise just db would be enough instead of db-migras.

Finally, start loading the media files (again, this is a slow process...)

ln -s ~/pk2_old/media_dump ~/parkour2/media_dump
make --directory ~/parkour2 load-media

Important

In the meantime, check that everything looks "normal" (e.g. compare to our parkour-dev mirror.) Once everything checks, we're ready to move forward.

make --directory ~/parkour2 save-postgres tar-old-migras
docker exec parkour2-rsnapshot rsnapshot halfy  ## only run this if load-media has finished!
date >> ~/last.txt
echo -e "SUCCESS\n\n" >> ~/last.txt
docker system prune -a -f
rm ~/parkour2/media_dump  ## clear symlink
mv ~/pk2_old/media_dump ~/parkour2/  ## move things over!

The End

Just a few last things that need to be taken care of...

• Edit the Pre-release, switch the checkboxes at bottom of that page, and make it the actual Latest release.
• Update parkour-demo (at least once per year.)
• If there was a new TLS certificate, check its expiration date and set a calendar reminder.
• parkour-dev VM is running on the previous release. Fix that: make down; git switch -; make dev-migras
• Develop new features, viel spaß 👍🏽