[BUG] Machine Tracker DNS searches eat all available file descriptors and crashes #2669
Comments
|
There may be problems reproducing this on systems that allow more than 1024 file descriptors per process. When running this in the docker-compose based developement environment, I'm unable to reproduce it, as the container (at least on my machine) seems to allow for much more than 1024 file descriptors. |
|
Inside the # ulimit -n
1048576This can be forced to more "normal" limits by adding something like the following to version: '2'
services:
web:
ulimits:
nofile:
soft: "1024"
hard: "1024" |
|
I don't think I put this in writing anywhere, but I guess the most likely course of action to resolve this is to impose some limit to how many simultaneous requests will be in-flight at any give time. E.g. an unlimited amount is untenable. 1024 would also be too high. 100 simultaneous requests might be a good (although arbitrary) number to ensure we don't bump against the usual limits. |
|
Fixed by #2917 |
Describe the bug
When running a Machine Tracker IP search for a subnet, while checking the
DNScheckbox, Machine Tracker will perform bunch of DNS lookups in parallel, in order to return the search results as fast as possible.However, it seems Machine Tracker parallelizes too much: Each resolver consumes a UDP socket, and the number of sockets uses seems to grow at least linearly with the size of the searched subnet.
Once the number of simultaneously open file descriptors reach the limit imposed by the OS (typically 1024), further attempts to open a file descriptor will crash with an unhandled
OSError, which in turn results in a series of chained errors, as Django tries to produce a 500 response, but is unable to open the500.htmltemplate because there are no more available file descriptors - and so Django tries to produce a500error for that error, and so on.The end result (at least when running NAV under
uwsgi) is that none of the opened file descriptors get closed, and the worker process stays around with too many open file descriptors. The worker process is now unable to respond properly to further requests, and all requests handled by it result in 500 Internal Server errors (until, in the case of ouruwsgisetup, when the worker process has been asked to handle a configured number of requests, theuwsgimaster process kills it and starts a new worker).To Reproduce
Steps to reproduce the behavior:
10.0.0.0/20in the IP address search fieldBothfilter radio buttonDnscheckbox in theColumnssection.Searchbutton.Expected behavior
Machine Tracker should respond in a timely manner for large subnet searches, just as with small ones, without crashing or leaking a bunch of resources.
Tracebacks
A typical traceback logged by a worker process that crashes may look like this (the exception chain is very long, so this is cut off as soon as it starts repeating itself):
Environment (please complete the following information):
Additional context
The open file descriptors of a
uwsgiprocess can be inspected using thelsofcommand line tool:The text was updated successfully, but these errors were encountered: