502 Bad Gateway error after upgrade

[nomad-cyanide]

Hi

We are running a prod and matching test-environment and I have been testing upgrading to version 3.6.9, but I I am having a problem with the upgrade (from 3.4.8 to 3.6.9). For starters everything looks to be running just fine, but then I run one of our longer running scripts, which collects a lot of data and put it into Netbox. This script works flawlessly on prod (3.4.8), but on the test-environment, running 3.6.9, with a copy of the prod-database and matching settings for the VM, it fails with the below message and it has not failed a the same place twice:

Traceback (most recent call last):
File "/prod/scripts/getfacts/ts_s2s_vpn_facts.py", line 394, in
main()
File "/prod/scripts/getfacts/ts_s2s_vpn_facts.py", line 309, in main
service_entry = nb_update_vs(nb_vs,sat_gw_ip,sat_gw_name,community_name,community_type,interop_hack_ip)
File "/prod/scripts/getfacts/ts_s2s_vpn_facts.py", line 124, in nb_update_vs
nb_service_entry_ip = str(nb_service_entry.ipaddresses[0]).split('/')[0]
File "/usr/local/lib/python3.10/dist-packages/pynetbox/core/response.py", line 327, in str
getattr(self, "name", None)
File "/usr/local/lib/python3.10/dist-packages/pynetbox/core/response.py", line 303, in getattr
if self.full_details():
File "/usr/local/lib/python3.10/dist-packages/pynetbox/core/response.py", line 459, in full_details
self._parse_values(next(req.get()))
File "/usr/local/lib/python3.10/dist-packages/pynetbox/core/query.py", line 291, in get
req = self._make_call(add_params=add_params)
File "/usr/local/lib/python3.10/dist-packages/pynetbox/core/query.py", line 258, in _make_call
raise RequestError(req)
pynetbox.core.query.RequestError: The request failed with code 502 Bad Gateway but more specific details were not returned in json. Check the NetBox Logs or investigate this exception's error attribute.

/var/log/nginx/error.log:

2024/01/08 14:27:35 [error] 919#919: *110908 recv() failed (104: Unknown error) while reading response header from upstream, client: 10.10.10.74, server: netbox-test.domain.test, request: "GET /api/ipam/ip-addresses/24202/ HTTP/1.1", upstream: "http://10.10.10.5:8001/api/ipam/ip-addresses/24202/", host: "netbox-test.domain.test"

Versions:
Netbox: 3.6.9
Python: 3.10.12
Redis version=6.0.16
nginx version: nginx/1.18.0 (Ubuntu)
psql (PostgreSQL) 14.10 (Ubuntu 14.10-0ubuntu0.22.04.1)
gunicorn (version 21.2.0)
pynetbox: 7.3.3

Linux vmnb02-test 5.15.0-91-generic #101-Ubuntu SMP Tue Nov 14 13:30:08 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Any help would be appreciated, otherwise I simply cannot justify upgrading.

[CTV-2023]

Maybe this ? #14691

[nomad-cyanide]

I thought so too, but how can I be sure?

[CTV-2023]

Test the workaround specified #14691 (comment) (note : needs to be tested in the venv)

[markkuleinio]

By reconfiguring max_requests to smaller value and running the tests again (does it fail sooner), maybe

[nomad-cyanide]

By reconfiguring max_requests to smaller value and running the tests again (does it fail sooner), maybe

Where would I correct this setting?

[markkuleinio]

/opt/netbox/gunicorn.py

Better discussion about the issue in here: benoitc/gunicorn#3038

[nomad-cyanide]

I have been testing a few things and this is my finding.
It doesn't help if I decrease the max_requests setting. It just fails sooner.
it doesn't help if I increase the max_requests setting either. It still fails at some point.
When it fails a log-entry about restarting shows up simultaneously:

Example:

an 09 13:20:07 30001vmnb02-test gunicorn[106082]: [2024-01-09 13:20:07 +0100] [106082] [INFO] Autorestarting worker after current request.
Jan 09 13:20:07 30001vmnb02-test gunicorn[106082]: [2024-01-09 13:20:07 +0100] [106082] [INFO] Worker exiting (pid: 106082)
Jan 09 13:20:08 30001vmnb02-test gunicorn[106256]: [2024-01-09 13:20:08 +0100] [106256] [INFO] Booting worker with pid: 106256

If I set the max_requests to 0, thus disabling it, my scripts work. without error. But is this preferable to having gunicorn processes restart regularly. I suppose it would start consuming memory, if it has memory-leak errors that is.

Perhaps a scheduled restart of the Netbox and netbox-rq services once a day would do the trick?

[markkuleinio]

Perhaps a scheduled restart of the Netbox and netbox-rq services once a day would do the trick?

I'm thinking about that (and max_requests 0) as well, if the memory leak is evident.

[nomad-cyanide]

Perhaps a scheduled restart of the Netbox and netbox-rq services once a day would do the trick?

I'm thinking about that (and max_requests 0) as well, if the memory leak is evident.

I have come to the conclusion, that rather than downgrade gunicorn of maybe loose some necessary features, I will go ahead with max_requests set i 0 and if memory usage becomes an issue on the server I will set up a scheduled job that restarts the worker processes with this command:

ps -aux | grep venv/bin/gunicorn | grep Ss | awk '{ system("kill -HUP " $2 )}'

[candlerb]

Simpler and safer, you can edit /etc/systemd/system/netbox.service to include

ExecStart=...as before...
ExecReload=/bin/kill -HUP $MAINPID

(remember systemctl daemon-reload after changing it), then your scheduled job can do systemctl reload netbox. (I expect there's a systemd timer way to do that too)