Broker fails to close used connections and they remain in CLOSE_WAIT

[kzangeli]

It seems, as discovered by a client, that the broker sometimes leaves used connections without closing them, causing all fds to be used and consequently the broker stops working.

The broker uses libmicrohttpd (MHD) to respond to requests and libcurl for notifications and forwarding of messages and it seems libcurl might be responsible for the detected problem.

[kzangeli]

Similar problem described in SOF:
http://stackoverflow.com/questions/12730477/close-is-not-closing-socket-properly

[fgalan]

I understand that the problem has been discoverd in Orion 0.26.1. Reviewing the changelog from Orion 0.26.1 to Orion 1.1 (the current one) note the following in Orion 0.28.0 release notes:

Fix: libmicrohttpd 0.9.48 included in contextBroker as static lib (previous Orion versions used 0.9.22 as dynamic library) (Issue #1675)

If the problem cause points to libmicrohttpd, it should be reviewed if 0.9.48 has solved it.

[crbrox]

Finally, it seems an expected behavior if the limit of file descriptors for the process is not high enough. With notification mode set as threadpool , each thread can hold up to 5 connections (libcurl pool). So, for outgoing request, 5*(number of pool's threads) file descriptors are needed to avoid exhausting the available fds

The total limit of fds should include the number of simultaneous incoming request in MHD, plus an amount for logs, listening sockets and file descriptors used by libraries
maxCon + 5 * no of threads + X

Or the number of threads in the pool can be lowered consequently

As Fermin has noted, the number of connections to mongoDB held by the driver should be included in the count
maxCon + 5 * no of threads + size of mongo driver pool + X

[fgalan]

After analyzing this, it seem it isn't a problem in the code, but something that need to be documented. Thus, removing "bug" label and adding "doc" label.

[fgalan]

Doc modifications in PR #2214

[mrutid]

It's happening...

[fgalan]

Recovering this issue for the 1.7.0 milestone.

[fgalan]

After researching on this, CLOSE_WAIT are not the root of the problem: the root of the problem is fd exahustion when select() was used in MHD (which is the scope of issue #2724).

However, documentation should be improved to explain this better so this issue will remain opened.

[fgalan]

Documentation completed in PR #2755. Closing.