Add support for Linux

[lewissbaker]

While most of cppcoro is platform agnostic, some of the thread-pooling and I/O code is OS-specific.

This issue is for providing an implemention of the cppcoro I/O abstractions for Linux.

Will need to get it building under Clang as a first step. See #3.
Then:

• Look into writing an epoll I/O event loop, based io_service implementation.
• Port file and socket (once written) interfaces to Linux.
• Add build-system support for Linux

[lewissbaker]

cppcoro is now at least buildable under Linux/Clang on the clang branch.

Still outstanding is the need to work on an implementation of io_service using epoll and then subsequent file/socket implementations.

[lewissbaker]

For async file I/O for io_service I'm thinking of using kernel aio (ie. io_submit, io_getevents and friends).
Although from my research this seems to be somewhat limited in that it's only truly async for files opened with O_DIRECT.

For async socket I/O it seems like epoll is the go.

Need to investigate how to mix the two approaches (aio/epoll) in the same event loop.

[neo5167]

So, did you guys start working on this?

[lewissbaker]

I haven't started on Linux async I/O support yet.

[neo5167]

I will work on it. But before I start can you help me compile the current code on linux? The build instructions aren't working for me.

[lewissbaker]

Sure. Can you create a new issue to work through the build issue?

[neo5167]

scheduler_operator_tests.cpp and io_service_tests.cpp are now working in linux. I will get file tests working as well and send a PR. I am using linux message queues as a sub for iocp, timerfd/eventfd for WaitableTimer/AutoresetEvent, and epoll for WaitForMultipleObjects. For files, I plan to send messages to the message queue from AIO's signal handling functions.

[lewissbaker]

Great! I look forward to the PR :)

For AIO, can you use an eventfd as the aio_resfd parameter of the iocb block for an I/O request as a way to have it wake up the epoll() loop when the operation completes. When epoll() indicates that the AIO eventfd is signalled then you know you need to call io_getevents() to retrieve pending I/O completion events. This should avoid the need for signal-handling functions.

Or are you talking about something different?

[neo5167]

That's right! I will try the resfd way. I remember a while ago that it wasn't stable. Its been long. They should've worked it out by now...

[lewissbaker]

@neo5167 How are you going with the epoll/aio io_service for Linux? Do you need any help?

[neo5167]

Was on vacation. How about I send a PR for everything without files and then send the files one once its done. WIN32 APIs are more prevalent in the files code than in timer/scheduler code, so taking longer than I expected...

[tavi-cacina]

Have you seen AFIO already? https://ned14.github.io/afio/
It states it supports coroutine TS (async_file_handle) and @ned14 seems very convinced he is on the right path (I watched few presentations from him).

[ned14]

I intend to propose AFIO as the File I/O TS in the June WG21 meeting. AFIO is the complete redesign after a Boost peer review in 2013, and therefore ought to carry some weight with regards to design with WG21 (but we shall see!). As you'll notice in AFIO, we don't prioritise async file i/o in the design, mainly because it isn't faster for most use cases, is definitely not faster on SSDs and is positively retrograde on SCM. I suspect you'll thus want to stick with your implementation. But keep at your library, looks valuable and I'd encourage you to submit it to Boost soon. We sorely need abstractions built on top of Coroutines both for Boost, and thence the C++ library.

[lewissbaker]

@neo5167 A PR with the basic io_service event loop and support for scheduling coroutines sounds like a great first step! I'd be happy to review it at that stage.
I've also been working on adding some socket APIs for Win32 in the sockets/sockets2 branch - this does some refactoring of the file_read/write_operation classes to eliminate some of the boiler-plate and also to add separate cancellable and non-cancellable variants of the operations (non-cancellable version generates much tighter assembly). Maybe check that out and see if a similar approach can be taken for the Linux implementations.

@tavi-cacina Thanks for pointing AFIO out. I had heard of it but haven't had a chance to look into it in any depth yet. I'll check it out.

@ned14 Thanks for the encouragement :) There's still a lot I want to add to it and a few things I want to revisit before I feel it will be baked enough for submitting to Boost.

[neo5167]

Great. Some info on timeline:
I work for Microsoft Research and we want to use cppcoro for some internal R&D on linux. I will share the PR as soon as our open source team approves the project. Should be less than a week. Thanks!

[neo5167]

Good to go on my end. Can I have write perms to create a branch to push my changes to? I can generate a PR after that.

[neo5167]

@lewissbaker ping. :)

[ericniebler]

@neo5167 The typical GitHub work flow is to create a fork of the repo, make your branch there, and make a pull request from there back to the original repo.

[neo5167]

Got it.

[lewissbaker]

@neo5167 My apologies for the delay in getting back to you. I've been travelling this week.

As @ericniebler suggested, it's probably best if you create your own fork of cppcoro and push your changes to a branch of that fork and then create a PR to merge those changes back into this repository.

I'm looking forward to seeing the PR, I think a lot of people are hanging out for it!

[neo5167]

@lewissbaker I created the PR.

[lewissbaker]

@neo5167 Great, I'll check it out.

[neo5167]

@lewissbaker Got a chance to try it out? I have a fixed few things. Will send another commit into the PR soon.

[lewissbaker]

@neo5167 I have started reviewing the code and am about 50% through.
I have had quite a number of other distractions the last couple of weeks, though, so it's taking a little longer than I'd hoped. I hope to finish the initial pass of the review soon.

[lni]

@lewissbaker @neo5167 thanks for the library and the port for Linux!

I grabbed a copy it and used it in the prototype application I am playing with. One issue I noticed is that the POSIX message queue based message_queue is pretty slow, it became a bottleneck when I need to do lots of io_service.schedule()/resume_on(). I plugged in Cameron Desrochers's lock free queue with built-in block queue wrapper, it solved my problem.

I guess this might be useful. Thanks.

[neo5167]

@lni Thanks for the input. I will take a look at that and see how we can use that.

[neo5167]

@lewissbaker Is it possible to get a private email address to chat with you? I wanted to go over somethings but prefer doing it in a private setting. Thanks!

[lewissbaker]

@neo5167 Sure, no worries. My email is my github username at gmail dot com.

[neo5167]

The linux scheduler and timer support code (beta) is at the following fork if anyone wants to try:
https://github.com/neo5167/cppcoro

[JasonDictos]

Hows linux support as of today?

[neo5167]

I did implement the linux support for an older version that is here:
https://github.com/neo5167/cppcoro

[JasonDictos]

There any important upstream changes in cppcoro branches that I'd be missing out on? It work? ;)

[JasonDictos]

oh btw one thing I noticed with your repo @neo5167 is the definition of the linux inner namespace conflicts with some other projects pre-processor definition of linux=1.

[JasonDictos]

Alright cool @neo5167 got it building with vcpkg, will report back if I find anything, here's the port file info if anyone is needing it: https://bitbucket.org/aparavi/vcpkg/src/aparavi/ports/cppcoro/

[dawidpilarski]

Hi there! Am I right to see, that the work on the Linux support is suspended right now (last comment on December 21st)? Any chance I can offer my help?

[Garcia6l20]

Hi there !
I just implemented the io_service, file and net APIs using liburing.
There are some requirements: GCC10 (10.2 fixes some compilation errors in lambda captures within a coroutine) and kernel >= 5.4 (5.5 for accept, connect, send, recv operations).
This is still a work-in-progress stuff but it is almost functional.

Check my fork.

Tell me what you think about it.

[Garcia6l20]

All tests passing !

make -j && ctest
[...]
100% tests passed, 0 tests failed out of 160

And my cppcoto-http extension works pretty well :)