[Upgrade Ubuntu] us_prod #965

dacook · 2024-11-21T00:26:13Z

Slack thread: #instance-managers

1. Setting up the new server

Check old server config for any additional services to be aware of. Document any necessary steps for migration. Eg:
- ls /etc/nginx/sites-enabled
- systemctl --state=running
Hosting: provision new server with Ubuntu 20
DNS: add temporary domain (eg prod2.openfoodnetwork.org)

config

Add temporary name to inventory/hosts
Review host_vars/x/config.yml, clean up if needed
- Make a copy for the temp hostname, add temp domain to bottom of certbot_domains
Review ofn-secrets:x_prod/secrets.yml, clean up if needed
- Change to shared bugsnag projects
- Don't bother making a copy of this one

setup

Enable passthrough on current server to allow new server to generate a certificate:

ansible-playbook playbooks/letsencrypt_proxy.yml -l x_prod -e "proxy_target=<new_ip>"

Then setup new server. Ensure you have the correct secrets (current secrets are usually fine).
ansible-playbook -l x_prod2 -e "@../ofn-secrets/x_prod/secrets.yml" playbooks/

setup.yml
provision.yml
deploy.yml
db_integrations (Permit DB access for n8n, Metabase)

initial migration

Ensure sidekiq is disabled, to avoid creating subscription orders when data is loaded:
sudo systemctl stop sidekiq && sudo systemctl disable sidekiq
Setup direct ssh access for ofn-admin and openfoodnetwork as per guide

ansible-playbook -l x_prod -e rsync_to=x_prod2 playbooks/

db_transfer.yml
transfer_assets.yml

Make sure to clear cache so that instance settings are applied:
cd ~/apps/openfoodnetwork/current; bin/rails runner -e production "Rails.cache.clear"

2. Testing

test reboot
send test mail (/admin/mail_methods/edit).
terms of service file: /admin/terms_of_service_files
shop catalogue display correctly, with images, add to cart, begin checkout, login
note: check cookies if login won't work
Check integrations
- Payments (check Stripe connect status /admin/stripe_connect_settings/edit)
- New Relic
- Bugsnag

3. Migration

preparation

Drop database openfoodnetwork on the new server: sudo -u postgres psql (Make sure you're on the new server!) To avoid any migration issues due to being out of sync, or jobs running in sidekiq.
Update ansible_host IP in inventory/hosts and ensure provision works (this should update host in .env.production and enable sidekiq again).
ansible-playbook playbooks/provision.yml -l x_prod
ansible-playbook playbooks/deploy.yml -l x_prod -e "git_version=vX.Y.Z" matching version with current prod
old server: make a tiny data change to verify later (eg add . in meta description /admin/general_settings/edit)

switchover: old server

🚧 ansible-playbook playbooks/maintenance_mode.yml -l x_prod
sudo systemctl stop sidekiq redis-jobs puma
ansible-playbook -l x_prod -e rsync_to=x_prod2 playbooks/db_transfer.yml &&
ansible-playbook -l x_prod -e rsync_to=x_prod2 playbooks/transfer_assets.yml
Transfer /var/lib/redis-jobs/dump.rdb to new server (see guide)
sudo systemctl stop postgresql (ensure other integrations no longer touch it)

switchover: new server

4. Cleanup (after 48hrs)

check the meet adn donate redirects still work
check server access logs to verify no traffic
shut down the old server,
cancel old VPS
~~check metabase sync if required: https://data.openfoodnetwork.org.uk/admin/databases/~~
check n8n
check backups are functioning

Rollback plan

If an error occurs before the temporary proxy is active, and can't be resolved quickly, then restore service back to current server
If an error occurs after proxy is active, users may have interacted with the new server (eg made payments.
- if serious, consider putting into maintenance mode (and stop sidekiq) to avoid further changes
- otherwise seek to resolve issue in-place.

The text was updated successfully, but these errors were encountered:

dacook · 2024-11-21T00:27:32Z

There are some subdomains pointing to the server (see Cloudflare DNS), but they appear to be simple redirects (probably set up in nginx)

meet.openfoodnetwork.net
donate.openfoodnetwork.net

lauriewayne · 2024-11-21T05:02:25Z

There are some subdomains pointing to the server (see Cloudflare DNS), but they appear to be simple redirects (probably set up in nginx)

meet.openfoodnetwork.net

donate.openfoodnetwork.net

Yep! Cloudflare gives us three redirects and we only use two. We use them a good amount.

dacook · 2024-12-10T23:20:51Z

todo: check backups are working after go-live.
https://app.bugsnag.com/yaycode/open-food-network-rails/errors/674687b441bcaf15eda5cc10?filters[error.status]=open&filters[event.since]=30d&filters[device.hostname]=us-prod2&pivot_tab=event

dacook · 2024-12-12T08:42:25Z

🏎️ US Prod server is upgraded! 🏁

Updated DNS, including "donate" and "meet" subdomains. These are actually proxied by cloudflare and redirected (see cloudflare Page Rules) before it ever reaches the server. But it seemed like a good idea to use the new IP instead of the old.

lauriewayne · 2024-12-12T14:25:51Z

Thank you David! You are the best!! One tiny non urgent question awaits in your email.

dacook · 2024-12-16T02:35:27Z

Checked access logs before shutting down old server. Just bots scanning for vulnerabilities (and.. guacamole?)

147.45.198.54 - - [15/Dec/2024:23:39:10 +0000] "GET /app_dev.php/_profiler/open?file=app/config/parameters.yml HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
147.45.198.54 - - [15/Dec/2024:23:39:11 +0000] "GET /_profiler/open?file=app/config/parameters.yml HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
147.45.198.54 - - [15/Dec/2024:23:39:12 +0000] "GET /app/config/parameters.yml HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
147.45.198.54 - - [15/Dec/2024:23:39:12 +0000] "GET /config/parameters.yml HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
147.45.198.54 - - [15/Dec/2024:23:39:13 +0000] "GET /parameters.yml HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
147.45.198.54 - - [15/Dec/2024:23:39:14 +0000] "GET /_profiler/phpinfo HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
147.45.198.54 - - [15/Dec/2024:23:39:14 +0000] "GET /app_dev.php/_profiler/phpinfo HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
147.45.198.54 - - [15/Dec/2024:23:39:15 +0000] "GET /phpinfo.php HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
147.45.198.54 - - [15/Dec/2024:23:39:16 +0000] "GET /owncloud/apps/graphapi/vendor/microsoft/microsoft-graph/tests/GetPhpInfo.php HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
147.45.198.54 - - [15/Dec/2024:23:39:16 +0000] "GET /info.php HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:105.0) Gecko/20100101 Firefox/105.0"
147.45.198.54 - - [15/Dec/2024:23:39:17 +0000] "GET / HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/105.0.0.0 Safari/537.36"
172.169.2.193 - - [15/Dec/2024:23:55:05 +0000] "GET /actuator/health HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 zgrab/0.x"
204.188.228.39 - - [16/Dec/2024:00:14:48 +0000] "GET /.env HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 Keydrop"
204.188.228.39 - - [16/Dec/2024:00:14:49 +0000] "GET /.env HTTP/1.1" 400 248 0.000 "-" "Mozilla/5.0 Keydrop"
167.94.146.55 - - [16/Dec/2024:00:38:47 +0000] "\x16\x03\x01\x00\xEE\x01\x00\x00\xEA\x03\x03\xB4\xF1\xCCQ\x856+\x02\x9D~\xE6\xD3\xEA/\xE8&" 400 150 0.328 "-" "-"
165.154.36.177 - - [16/Dec/2024:00:59:14 +0000] "GET / HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0"
165.154.36.177 - - [16/Dec/2024:00:59:32 +0000] "GET /favicon.ico HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
165.154.36.177 - - [16/Dec/2024:00:59:32 +0000] "GET /robots.txt HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
165.154.36.177 - - [16/Dec/2024:00:59:32 +0000] "GET /sitemap.xml HTTP/1.1" 301 162 0.000 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_0) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.56 Safari/535.11"
5.188.118.216 - - [16/Dec/2024:01:20:36 +0000] "GET / HTTP/1.1" 301 162 0.000 "-" "python-requests/2.32.3"
5.188.118.216 - - [16/Dec/2024:01:20:37 +0000] "GET /guacamole HTTP/1.1" 301 162 0.000 "-" "python-requests/2.32.3"

I hope they find what they're looking for.

dacook · 2024-12-16T02:39:33Z

Hi @lauriewayne just confirming I have shutdown the old server 🖥️ 💤 . You can delete it when you're ready.

lauriewayne · 2024-12-16T02:53:38Z

Hi @lauriewayne just confirming I have shutdown the old server 🖥️ 💤 . You can delete it when you're ready.

We will have a moment of gratitude to the old server for its years of service, and to you @dacook or helping us safely move into the future! Then that old droplet is gone! 💧 👋 🙏 (I will also remove that bowl of guacamole we left laying around after the last party) 🥑

dacook mentioned this issue Nov 21, 2024

Upgrade Ubuntu #157

Closed

9 tasks

github-project-automation bot added this to OFN Delivery board Nov 21, 2024

github-project-automation bot moved this to All the things 💤 in OFN Delivery board Nov 21, 2024

dacook changed the title ~~us_prod~~ [Upgrade Ubuntu] us_prod Nov 21, 2024

dacook self-assigned this Nov 21, 2024

dacook moved this from All the things 💤 to In Progress ⚙ in OFN Delivery board Nov 21, 2024

dacook mentioned this issue Dec 12, 2024

US prod #971

Merged

dacook closed this as completed Dec 16, 2024

github-project-automation bot moved this from In Progress ⚙ to Done in OFN Delivery board Dec 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Upgrade Ubuntu] us_prod #965

[Upgrade Ubuntu] us_prod #965

dacook commented Nov 21, 2024 •

edited

Loading

dacook commented Nov 21, 2024 •

edited

Loading

lauriewayne commented Nov 21, 2024

dacook commented Dec 10, 2024

dacook commented Dec 12, 2024 •

edited

Loading

lauriewayne commented Dec 12, 2024

dacook commented Dec 16, 2024

dacook commented Dec 16, 2024

lauriewayne commented Dec 16, 2024 •

edited

Loading

[Upgrade Ubuntu] us_prod #965

[Upgrade Ubuntu] us_prod #965

Comments

dacook commented Nov 21, 2024 • edited Loading

1. Setting up the new server

config

setup

initial migration

2. Testing

3. Migration

preparation

switchover: old server

switchover: new server

4. Cleanup (after 48hrs)

Rollback plan

dacook commented Nov 21, 2024 • edited Loading

lauriewayne commented Nov 21, 2024

dacook commented Dec 10, 2024

dacook commented Dec 12, 2024 • edited Loading

lauriewayne commented Dec 12, 2024

dacook commented Dec 16, 2024

dacook commented Dec 16, 2024

lauriewayne commented Dec 16, 2024 • edited Loading

dacook commented Nov 21, 2024 •

edited

Loading

dacook commented Nov 21, 2024 •

edited

Loading

dacook commented Dec 12, 2024 •

edited

Loading

lauriewayne commented Dec 16, 2024 •

edited

Loading