Add "die-on-term" to fix uWSGI shutdown

[markkuleinio]

Change Type

Addition

Area

Installation/upgrade

Proposed Changes

When using "systemctl stop netbox", systemd sends SIGTERM to the process (see https://www.freedesktop.org/software/systemd/man/latest/systemd.kill.html#KillSignal=).

However, as documented in https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi, for uWSGI the SIGTERM signal means:

brutally reload all the workers and the master process

= it is not a "stop" signal for uWSGI.

(... "this bad choice has been fixed in uWSGI 2.1" as mentioned by the maintainer in https://uwsgi-docs.readthedocs.io/en/latest/ThingsToKnow.html, but it is apparently not happening any time soon.)

Thus, I'm proposing adding die-on-term = true to be added in the NetBox-default uwsgi.ini.

That makes the default systemctl behavior to work with uWSGI as well, instead of changing the kill signal in netbox.service, to keep it simplier for also Gunicorn use.

With that addition NetBox shuts down correctly with "systemctl stop netbox":

May 08 18:42:53 netbox-lab systemd[1]: Stopping netbox.service - NetBox WSGI Service...
May 08 18:42:53 netbox-lab uwsgi[572422]: SIGINT/SIGTERM received...killing workers...
May 08 18:42:54 netbox-lab uwsgi[572422]: worker 1 buried after 1 seconds
May 08 18:42:54 netbox-lab uwsgi[572422]: goodbye to uWSGI.
May 08 18:42:54 netbox-lab systemd[1]: netbox.service: Deactivated successfully.
May 08 18:42:54 netbox-lab systemd[1]: Stopped netbox.service - NetBox WSGI Service.

---

Note: With the current default configuration, "systemctl stop netbox" just happens to currently work, because:

May 08 18:33:06 netbox-lab systemd[1]: Stopping netbox.service - NetBox WSGI Service...
May 08 18:33:06 netbox-lab uwsgi[572209]: ...brutally killing workers...
May 08 18:33:07 netbox-lab uwsgi[572209]: worker 1 buried after 1 seconds
May 08 18:33:07 netbox-lab uwsgi[572209]: binary reloading uWSGI...
May 08 18:33:07 netbox-lab uwsgi[572209]: chdir() to /opt/netbox
May 08 18:33:07 netbox-lab uwsgi[572209]: closing all non-uwsgi socket fds > 2 (max_fd = 1024)...
May 08 18:33:07 netbox-lab uwsgi[572209]: found fd 3 mapped to socket 0 (127.0.0.1:8001)
May 08 18:33:07 netbox-lab uwsgi[572209]: running /usr/bin/python3.11
May 08 18:33:07 netbox-lab uwsgi[572209]: Traceback (most recent call last):
May 08 18:33:07 netbox-lab uwsgi[572209]:   File "/opt/netbox/venv/bin/uwsgi", line 5, in <module>
May 08 18:33:07 netbox-lab uwsgi[572209]:     from pyuwsgi import run
May 08 18:33:07 netbox-lab uwsgi[572209]: ModuleNotFoundError: No module named 'pyuwsgi'
May 08 18:33:07 netbox-lab systemd[1]: netbox.service: Main process exited, code=exited, status=1/FAILURE
May 08 18:33:07 netbox-lab systemd[1]: netbox.service: Failed with result 'exit-code'.
May 08 18:33:07 netbox-lab systemd[1]: Stopped netbox.service - NetBox WSGI Service.

= it tries to reload uWSGI but a module import error prevents that happening.

I don't currently know why that happens, I haven't used pyuwsgi before, and this module import problem does not occur with uwsgi package. (But the "reload vs. stop" problem is still valid with uwsgi as well, so die-on-term = true is needed anyway.)

[markkuleinio]

I'm willing to provide the PR for this change.

[markkuleinio]

Writing this down here:

With pyuwsgi:

May 08 18:55:49 netbox-lab uwsgi[572660]: *** Starting uWSGI 2.0.23 (64bit) on [Wed May  8 18:55:49 2024] ***
May 08 18:55:49 netbox-lab uwsgi[572660]: detected binary path: /usr/bin/python3.11

With uwsgi:

May 08 19:02:59 netbox-lab uwsgi[573360]: *** Starting uWSGI 2.0.25.1 (64bit) on [Wed May  8 19:02:59 2024] ***
May 08 19:02:59 netbox-lab uwsgi[573360]: detected binary path: /opt/netbox/venv/bin/uwsgi

Looks like pyuwsgi somehow finds itself to be the system-default Python binary, instead of discovering the virtual environment, and maybe thus the module import fails whenever a reload is attempted.

I didn't spot any relevant commit diffs between uWSGI 2.0.23 (which latest pyuwsgi still uses) and 2.0.25.1.

[DanSheps]

However, as documented in https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi, for uWSGI the SIGTERM signal means:

Can I just say how utterly stupid that is since SIGTERM is also the default signal from kill...

[jeremystretch]

Fun stuff. All yours @markkuleinio. Thanks for staying on top of these uWSGI items!

[markkuleinio]

FYI, lincolnloop/pyuwsgi-wheels#23 opened about the pyuwsgi reload problem.