POSIX pollqs should scale horizontally

[gdamore]

Right now we are using only a single thread for POSIX based pollqs.

There are some reasons for this, not the last of which is that event removal is difficult to coordinate across different threads. But we can either scale to multiple pollqs (based on nni_plat_ncpu), or examine the opportunity to scale a single pollq to multiple threads.

Specially the poll based poller (which nobody uses) should expand to multiple threads trivially. The others will be easier or harder depending on how we can synchronize them.

[zhuoyw]

Are there any plans for this feature? One pollq thread is not enough to leverage line rate.

[gdamore]

It should not be hard to do. Just haven’t got around to it.

[gdamore]

Which platform do you need this for ?

[gdamore]

There is quite a lot of evidence that this wasn't the main performance limiter. I'm moving this to 1.5. (Other issues, like the aio lock being highly contended, were a higher priority.)

[gdamore]

So I have recently been in this code, and while i don't really want to do this for the select poller, the other pollers can do it pretty much nearly trivially. The epoll() one is probably the highest priority, followed by kqueue().