[Upgrade Ubuntu] ie_prod

[dacook]

The upgrade has already been performed by Cillian (see #951), but I noticed it is missing some config (eg New Relic) so we should run provision.yml to ensure all details are set.
Maybe double-check the details in ofn-secrets first to check they match.

[dacook]

FYI @cillian , I am going to run the provision.yml playbook to ensure our config is in sync, and the monitoring service (New Relic) is set up.
edit: For context, we retain all "secrets" such as those in .env.production in a protected repository called ofn-secrets. Whenever we provision the server, it will overwrite the .env.production file with those secrets, so it's necessary for them to be in sync.

I also note that it is set up to send Bugsnag reports to project that we don't have access to.
Do you require this, or would it be ok for us to send bugsnag reports to our shared project so that we can analyse bigger trends?

[cillian]

@dacook Okay thanks, let me know if you need me to do anything. Sending BugSnag reports to your shared project should be fine, although can we still have access too, for debugging?

[dacook]

Thanks Cillian. Actually I'm going to backtrack; we seem to be hitting limits on our Bugsnag account, and I haven't resolved yet about providing broader access to the shared project (I think it's probable that it contains personal data from other instances).
So for now, I'll leave bugsnag as-is until we can resolve those things sometime in the future.

[dacook]

I've run the provision script, with an excerpt of changes below for your info. Very large log ensues...

~/projects/ofn-install $ ansible-playbook playbooks/provision.yml --limit=ie_prod -e "@../ofn-secrets/ie_prod/secrets.yml"; beep
TASK [app_user : Write OFN environment variables defaults] ****************************************************************************

changed: [openfoodnetwork.ie]

Thursday 07 November 2024  09:44:38 +1100 (0:00:14.456)       0:04:30.704 *****
TASK [app_user : Write bash_profile for app user] *************************************************************************************

changed: [openfoodnetwork.ie]

Thursday 07 November 2024  09:44:52 +1100 (0:00:14.247)       0:04:44.951 *****
TASK [app_user : Create sudoers configuration for app user] ***************************************************************************

changed: [openfoodnetwork.ie]

Thursday 07 November 2024  09:45:05 +1100 (0:00:13.198)       0:04:58.150 *****
TASK [config : Set hostname] **********************************************************************************************************

changed: [openfoodnetwork.ie]

Thursday 07 November 2024  09:45:12 +1100 (0:00:07.058)       0:05:05.208 *****
TASK [app : template files] ***********************************************************************************************************

changed: [openfoodnetwork.ie] => (item={'src': 'env.j2', 'dest': '/home/openfoodnetwork/apps/openfoodnetwork/shared/config/.env.production'})

changed: [openfoodnetwork.ie] => (item={'src': 'postgresql.yml.j2', 'dest': '/home/openfoodnetwork/apps/openfoodnetwork/shared/config/database.yml'})

Thursday 07 November 2024  09:49:56 +1100 (0:00:26.262)       0:09:48.675 *****
TASK [node : install nodenv plugins] **************************************************************************************************

changed: [openfoodnetwork.ie] => (item={'name': 'node-build', 'repo': 'https://github.com/nodenv/node-build.git', 'version': 'master'})
TASK [node : add nodenv to user path] *************************************************************************************************

changed: [openfoodnetwork.ie]
TASK [geerlingguy.postgresql : Ensure PostgreSQL users are present.] ******************************************************************

changed: [openfoodnetwork.ie] => (item=None)

changed: [openfoodnetwork.ie]
TASK [dbserver : add .pgpass file for openfoodnetwork] ********************************************************************************

changed: [openfoodnetwork.ie]
TASK [coopdevs.certbot_nginx : Extract current domains list from the certificate] *****************************************************

[WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo

changed: [openfoodnetwork.ie]
TASK [coopdevs.certbot_nginx : Force generation of a new certificate] *****************************************************************

changed: [openfoodnetwork.ie]
TASK [jdauphant.nginx : Create the configurations for sites] **************************************************************************

ok: [openfoodnetwork.ie] => (item={'key': 'default', 'value': ['listen 80 default_server;\nlisten [::]:80 default_server;\nserver_name  _;\nadd_header X-Content-Type-Options nosniff always;\nadd_header X-Xss-Protection "1; mode=block" always;\nadd_header X-Frame-Options DENY always;\nadd_header Content-Security-Policy "default-src none" always;\nlocation / {\n  access_log off;\n  return 444;\n}\n']})

changed: [openfoodnetwork.ie] => (item={'key': 'ofn_80', 'value': ['listen 80;\nlisten [::]:80;\nserver_name openfoodnetwork.ie www.openfoodnetwork.ie;\n\nadd_header X-Content-Type-Options nosniff always;\nadd_header X-Xss-Protection "1; mode=block" always;\n\nlocation '/.well-known/acme-challenge' {\n  default_type "text/plain";\n  root /etc/letsencrypt/webrootauth;\n}\n\n\n\nlocation / {\n  limit_except GET POST PUT PATCH DELETE OPTIONS { deny all; }\n  return 301 https://openfoodnetwork.ie$request_uri;\n}\n']})

changed: [openfoodnetwork.ie] => (item={'key': '000_redirect_www', 'value': ['listen 443 ssl http2;\nlisten [::]:443 ssl http2;\nserver_name www.openfoodnetwork.ie;\n\nssl_certificate      /etc/letsencrypt/live/openfoodnetwork.ie/fullchain.pem;\nssl_certificate_key  /etc/letsencrypt/live/openfoodnetwork.ie/privkey.pem;\n\nssl_protocols TLSv1.2;\nssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA;\nssl_prefer_server_ciphers on;\n\n\n\nreturn 301 https://openfoodnetwork.ie$request_uri;\n']})

changed: [openfoodnetwork.ie] => (item={'key': 'ofn_443', 'value': ['listen 443 ssl http2;\nlisten [::]:443 ssl http2;\nserver_name openfoodnetwork.ie www.openfoodnetwork.ie;\nroot /home/openfoodnetwork/apps/openfoodnetwork/current/public;\n\nssl_certificate      /etc/letsencrypt/live/openfoodnetwork.ie/fullchain.pem;\nssl_certificate_key  /etc/letsencrypt/live/openfoodnetwork.ie/privkey.pem;\n\nssl_protocols TLSv1.2;\nssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA;\nssl_prefer_server_ciphers on;\n\n\n\nadd_header X-Content-Type-Options nosniff always;\nadd_header X-Xss-Protection "1; mode=block" always;\n\ngzip on;\ngzip_types text/css text/javascript text/plain application/javascript application/x-javascript application/json;\ngzip_disable "msie6";\n\nbrotli on;\nbrotli_types text/css text/javascript text/plain application/javascript application/x-javascript application/json;\n\ntry_files $uri/index.html $uri @rails;\nlocation @rails {\n  limit_except GET POST PUT PATCH DELETE OPTIONS { deny all; }\n\n  if (-f /etc/nginx/maintenance.html) {\n    return 503;\n  }\n\n  gzip_proxied no-cache no-store private expired auth;\n  proxy_http_version 1.1;\n  proxy_set_header X-Real-IP $remote_addr;\n  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;\n  proxy_set_header Host $host;\n  proxy_set_header X-Forwarded-Proto $scheme;\n  proxy_set_header X-Request-Start "t=${msec}";\n  proxy_redirect off;\n  proxy_pass http://rails;\n}\n\nlocation ~ ^/(assets)/ {\n  limit_except GET POST PUT PATCH DELETE OPTIONS { deny all; }\n  gzip_static on;\n  brotli_static on;\n  expires max;\n  add_header Cache-Control public;\n}\n\nerror_page 500 502 504 /500.html;\nerror_page 503 @maintenance;\n\nlocation @maintenance {\n  limit_except GET POST PUT PATCH DELETE OPTIONS { deny all; }\n  root /etc/nginx;\n  try_files /maintenance.html =503;\n}\n\nlocation /cable {\n  limit_except GET POST PUT PATCH DELETE OPTIONS { deny all; }\n  proxy_pass http://rails;\n  proxy_http_version 1.1;\n  proxy_set_header X-Forwarded-Proto https;\n  proxy_set_header X-Forwarded-Ssl on;\n  proxy_set_header Upgrade $http_upgrade;\n  proxy_set_header Connection "upgrade";\n  proxy_set_header Host $host;\n}\n\nclient_max_body_size 4G;\nkeepalive_timeout 30;\nproxy_read_timeout 30;\nproxy_send_timeout 30;\n\ninclude /etc/nginx/sites-available/ofn/*;\n']})

Thursday 07 November 2024  10:11:13 +1100 (0:00:48.045)       0:10:46.661 *****
TASK [newrelic : Download installer] **************************************************************************************************

changed: [openfoodnetwork.ie]

Thursday 07 November 2024  10:13:12 +1100 (0:00:06.368)       0:12:45.523 *****
TASK [newrelic : Install New Relic command] *******************************************************************************************

changed: [openfoodnetwork.ie]

Thursday 07 November 2024  10:13:18 +1100 (0:00:06.120)       0:12:51.644 *****
TASK [newrelic : Install New Relic agent] *********************************************************************************************

changed: [openfoodnetwork.ie]

Thursday 07 November 2024  10:14:52 +1100 (0:01:34.568)       0:14:26.212 *****
TASK [newrelic : Use our standard names in dashboard] *********************************************************************************

changed: [openfoodnetwork.ie]

Thursday 07 November 2024  10:14:58 +1100 (0:00:05.902)       0:14:32.115 *****
RUNNING HANDLER [jdauphant.nginx : reload nginx - after config check] *****************************************************************

changed: [openfoodnetwork.ie]

Thursday 07 November 2024  10:15:24 +1100 (0:00:05.983)       0:14:58.086 *****

(BTW There was an issue with updating node-build with git, which I manually corrected with git on the server)

Sorry that we don't have more transparency around the ofn-secrets file. This doesn't need to be hidden from you, but we haven't got a good system in place for full transparency and collaboration right now. I have some thoughts on improving that but don't have time right now..

🚨 But server is now down... investigating...

[dacook]

✅ fixed. The provision script had updated the database password, but didn't restart the puma process. I've manually restarted puma and sidekiq and it looks mostly good.

However I notice the logo at top-left of the page (desktop only) is not loading, as it has the temporary URL (ubuntu-20). I'm not sure if there will be other assets having that URL, so I've solved it with an nginx redirect with https cert for the temporary domain.

All looks ok to me now, but let me know if you spot anything else.

[dacook]

New Relic is working.
Cillian, if you send me your email address, I think I might be able to grant you access.

[cillian]

@dacook No problem about ofn-secrets and thanks for setting up Ngninx and fixing the logo. I re-uploaded those logos and they now seem to give a 302 redirect, instead of the 301 from the nginx redirect you set up. I wonder if they were pointing at ubuntu-20.openfoodnetwork.ie because I had to upload them manually during the server migration, this was before I noticed /storage wasn't copied over and before switching DNS.

[dacook]

Thanks Cillian, I've sent an invite for NR.
I think the 302 redirects are expected as part of the ActiveStorage implementation.

[dacook]

I've run the db_integrations.yml playbook now, which created database users for metabase and n8n.