Add kqueue support for Darwin/BSD

[gdamore]

We would like to have kqueue based polling for improved scalability.

[liamstask]

Hi - I began taking a look at this since it looked like a reasonably well isolated task, and had a couple questions about your vision for the polling system.

Currently, the poll based poller is assumed for all posix platforms - it's enabled in posix_config.h and used throughout the posix platform code.

Since some posix platforms will want to use different polling implementations (kqueue, epoll, etc), do you imagine creating a polling api that would be implemented for those backends? Maybe you have another approach in mind?

Let me know and I can either try to assist or just stay out of the way until it's clearer how to proceed 😄

[gdamore]

Sorry I didn't respond earlier. Thanks for your interest in contributing to nng!

So I think you can look at the posix_pollq_poll.c to see how to write a poller. There is a common API. I'm not 100% certain that it is fully adequate, but if you find you need something that the core doesn't provide, let me know and we'll work it through. Fundamentally, you have to implement these functions:

nni_posix_pollq_add() - adds a pollq node to the poller
nni_posix_pollq_remove() - removes a pollq node from the poller (usually just before it is destroyed)
nni_posix_pollq_arm() - arms events (which are poll(2) event items like POLLIN) on the node
nni_posix_pollq_disarm() - disarms events
nni_posix_pollq_get() - obtain a "pollq" for a given file descriptor. The idea here was to support multiple pollq's for different CPUs, to get some cache coherence. Right now I'm just returning a single global one.

When events that have been armed occur on the file descriptor associated with the node, you need to set the "revents" member and call the user supplied callback (cb). Reading the posix_pollq_poll.c code should help you see what needs to be done.

The way I'd like to see the use of kqueue and the like enabled is via CMake tests, for example look at how we detect the availability of clock_gettime() in the top level CMakeLists.txt. This needs to enable a different NNG_USE_POSIX_POLLQ_KQUEUE variable, and the setting of the current in posix_config.h needs to be made conditional, much like the clock definitions above it are done.

I had btw started doing this for epoll(), but set it aside for a bit. I seem to recall that I did need to do something different there, and that I had some additional state requirements that had to be added to pollq_node, but I don't remember exactly what they were.

[gdamore]

Oh, you may need to implement the functions nni_posix_pollq_sysinit() and nni_posix_pollq_sysfini(). See the posix_pollq.h header file -- that is actually intended to "formally" define the polling API for the backend implementations on UNIX-like systems.

[liamstask]

Got it. It seems a little strange to depend on poll(2) event symbols (POLLIN, etc) while these other systems do not otherwise rely on poll(2) - is the idea to try to reuse those in order to avoid creating new symbols? Seems a little cleaner to avoid including poll.h altogether on platforms that won't use it, but probably not a huge deal.

Otherwise, understood that posix_pollq.h is meant to be the general api for polling on posix systems 👍

[gdamore]

So the reason we rely on those symbols (POLLIN, POLLOUT, etc.) is that they are required to be defined on POSIX systems. We could have invented our own symbols for this, but there seemed to be little point in doing so, and by using the symbols that are well known, we actually can simplify some of the backends. (For example, the poll(), epoll(), and /dev/poll implementations all use the same symbols, or symbols that are guaranteed to be numerically equivalent.) kqueue is the odd one our here, I think, since its EVFILT stuff is quite a bit different.

[gdamore]

Fixed in 7849fd2