RFC 0002: MT Execution Contexts

[ysbaddaden]

No description provided.

[beta-ziliani]

Preview: https://github.com/crystal-lang/rfcs/blob/rfc-0002-mt-execution-contexts/text/0002-execution-contexts.md

[straight-shoota]

The distinction between execution context and scheduler could need a bit refinement. There is definitely some overlap in functionality, just by comparing the API. I guess execution contexts might take over some features of the current scheduler?

[RX14]

Absolutely lovely, well-written proposal. I completely agree with the design intent here, and only have a few—mostly overlapping—comments about event loops and the default context. The vast majority of this design is exactly what I would like to see in crystal.

[RX14]

Are multiple event bases required? It doesn't seem obvious to me that they are.

[ysbaddaden]

Yeah, I'm wondering about that. We currently have one libevent base per thread, and that overcomplexifies IO::Evented with lots of objects to allocate in the HEAP for each IO and thread.

I'm probably being naive (though Go seems to do that) but maybe a global event loop wouldn't behave so badly? Even with the potential contention on adding an event to the libevent base, especially on machines with too many cores to count (e.g. ARM Neoverse).

[yxhuvud]

On the other hand not having to synchronize everything and their mother make other things a lot less complex. It is quite liberating to not give a shit about what other threads are doing.

FWIW, having one dedicated ring per thread is also how the makers of io_uring recommend multi thread usage.

[ysbaddaden]

Hum, implementing our own wrapper on top of epoll/kqueue becomes more and more compelling.

Since we reschedule the fiber, we could put events on the stack —this is possible with libevent but not recommended: the struct size may change, and not have to keep them somewhere to try and avoid reallocating events all the time.

Then we could have one or many event loops and not care about thread locals (especially in IO); we could keep Fiber#resume_event and merely take care that it can only be in one event loop queue at a time.

[yxhuvud]

Yes, storing it on the stack is a very useful technique for io_uring as well (to keep it alive during submission). I've made a lot of use of that and it helped a lot. There the completion event need to be handled somehow though - I stored it in the fiber itself until it was awakened and could process it, but there may be better techniques.

One good thing about the current implementation though is that it handles the thundering herd problem decently, as far as I've been able to see. That is, if multiple fibers are waiting for something only one of them will wake when something happens. With many listeners that may become something to keep track of.

[ysbaddaden]

We shall move the "refactor the event loop" into a proper issue on https://github.com/crystal-lang/crystal/issues

It doesn't need a RFC as it's mostly internal implementation detail.

[ysbaddaden]

It might be interesting to consider one event loop (EL) per execution context (EC) 🤔

With a single EL per EC, a starving scheduler will run the EL and enqueue every resumable fibers for the EC.

With one EL per scheduler, a starving scheduler will run its own EL and push the fraction of fibers it happened to have, possibly flooding the EC queues and delaying other schedulers from running their own EL, and delaying resumable fibers from being resumed.

That may be a reason why Go has a single EL: it could be unfair otherwise.

[RX14]

This proposal supports creating multiple execution contexts, let the application configure it's own EC and start fibers in it if required. That allows all the complexity the app needs when configuring the context the application actually runs in, because it's initialized by the application. The root context does not have to be well-used by the application.

[ysbaddaden]

@RX14 Tell me if I'm wrong, the plan would be:

Crystal 1

• introduce EC with ST and MT;
• deprecate same_thread argument;
• same_thread: false is NOOP;
• ST accepts same_thread: true (always true anyway);
• MT raises on same_thread: true (new API, no breaking change);
• default EC is ST (no breaking change);
• consider a -Dmt flag to force default EC to be MT (?);

Crystal 2

• remove deprecated same_thread (breaking change);
• default EC becomes MT (breaking change).

Isolated context

I think I see that context for UI loops only, and want to prevent blocking behaviors, but there's nothing wrong with doing blocking calls in other use cases, and using the event-loop normally is fine. Still, spawning a fiber without an explicit context should either raise or the default context should be configured (as you suggest):

abstract class ExecutionContext
  class Isolated < ExecutionContext
    def initialize(name : String, @spawn_context : ExecutionContext? = nil, &)
      @thread = Thread.new(name) { yield }
    end

    def spawn(**args, &) : Fiber
      if ctx = @spawn_context
        ctx.spawn(**args) { yield }
      else
        raise RuntimeError.new("Can't spawn in isolated context (need a spawn context)")
      end
    end
  end
end

mt = ExecutionContext::MultiThreaded.new
ui = ExecutionContext::Isolated.new("GTK", spawn_context: mt) { Gtk.main }

Instead of raising, the spawn context could be the default EC.

[ysbaddaden]

@RX14 I applied your suggestions to the RFC.

[RX14]

@ysbaddaden I think -Dmt is probably not necessary, and the exact implementation plan for crystal 2 is best left deferred until there's operational experience, but I agree on everything else.

I envision the root execution context being MT or ST a moot point, because every well-architected app has a single App.run line at the top-level and converting that top-level code to be spawning a MT context and waiting for that fiber should be a one-liner if we have the right helper methods in place.

If we all agree, maybe we can start on the other 90% of the RFC: bikeshedding naming. I like ExecutionContext::Parallel, because I don't like the idea of implementation details (threads) leaking into the name.

[ysbaddaden]

@RX14 The mt flag may not be necessary in Crystal v1, as the default context could be MT:1 and resized on demand (still no breaking change). I'll still push for MT:N to be the default in Crystal v2. Execution contexts are a mean to further control the parallelism in very specific cases, not the end solution. I believe developers shouldn't have to care about it until you have to.

I wouldn't bikeshed the namings just yet. As I'm experimenting with the types, I feel that the difference is getting thinner and thinner. In fact, Kotlin only has a single scheduler implementation, and a couple constructors to start execution contexts with 1 (ST) or many threads (MT).

I'm also struggling with the inheritance: EC::MT < EC makes sense, but so does EC::MT::Scheduler < EC as we want EC.current to point to the current MT scheduler running on the thread, not the shared MT context (it's easier to reach the context from the scheduler).

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/crystal-multithreading-support/6622/8

[ysbaddaden]

I forgot again but MT:1 would break spawn(same_thread: true) in Crystal 1. It's a NOOP without the preview_mt flag but the parameter was still exposed to the public API 😭

[straight-shoota]

I think we can accept breakage with same_thread: true. It only works with preview_mt which is explicitly a preview feature. There should be no expectations on compatibility in a setting outside of preview_mt.

[yxhuvud]

I like this, I like this a lot.

[yxhuvud]

I think data and especially execution locality could show up on the negative side as well, as the round robin take away a lot of programmatic control of data locality as well. It is .. possible.. to manually schedule fibers to dedicated threads but it really is not the way it currently is meant to be used.

[yxhuvud]

I'm in the 'let the event loop decide if it want to be instantiated on a thread, execution level or global level' camp. How that would look API-wise I'm less sure - especially not if dynamic amount of threads in a context is to be supported.

This might be complex: I don’t think we can share a libevent across event bases

From what I have gathered from libevent docs, it is possible, but would necessitate a lot more synchronization when io happens (*), so it is probably slower.

But yes, it is complex. Windows, and its weird file handles says hi. Each open file handle is specific for each instance of whatever it uses, so it needs to be only one global one event instance there.

• we already enable some structures for thread safety but then create separate bases for each thread anyhow, IIRC. It was quite a while since I looked at it. I think we can remove that enabling without danger - they should really only be used when actually reusing a libevent base between threads). We don't use the specialized mt safe functions libevent that make use of it.

[yxhuvud]

It probably needs to be mentioned that it needs to continue to work with channels and mutexes. This is somewhat straightforward today with how fibers are bound to a thread once executed, so the simple version is to just schedule it using the normal interfaces. But interfaces for scheduling are not necessarily the same between execution contexts as they are within it!

For example, a basic work stealing scheduler can default to just enqueue a fiber in the executing scheduler/thread context, and let distribution between threads happen in other ways. But that doesn't work if the thing to wake up doesn't live in the same context - then it needs to be communicated somehow. And then the question is what to communicate it to.

Also somewhere it should probably be explicitly defined what happens if channel interaction happens in an isolated context (as defined above).

[ysbaddaden]

This is explicit in the Guide side:

Applications can create any number of execution contexts in parallel. These contexts are isolated but they shall still be capable to communicate together with the usual synchronization primitives (e.g. Channel, Mutex) that must be thread-safe.

Once spawned a fiber shouldn’t move to another execution context. For example on re-enqueue the fiber must be resumed into it’s execution context: context B enqueues waiting sender from context A. That being said, we could allow to send a fiber to another context.

It's not detailed in the Technical side, though.

[yxhuvud]

Allowing a dynamic amount of threads requires more synchronization and complexity than having a static amount. While it sounds nice to be able to adjust, it probably warrants its own separate class. Making certain all threads are in a waiting state before starting to actually queue stuff allows a bunch of simplifications with less mutexes and risks a lot fewer possible race conditions too.

[ysbaddaden]

Enqueue doesn't have to need much sync. Go pushes to a bounded local queue (per scheduler) with overflow to a global queue; threads can be started at any time: when they reach the run loop they will grab a batch of fibers from the global queue or steal from another scheduler. Stopping ain't more complex, schedulers aren't tied to a specific thread, the thread detaches the scheduler and returns itself to the thread pool.

The complexity is more in when to start / stop a thread.

[ysbaddaden]

And that complexity pushes for it to become a "future evolution".

[Blacksmoke16]

As someone who isn't familiar with this stuff at all, my random question is:

Do we need to do anything in relation to like how Intel has P and E cores now? Like as a way to signal to the OS's thread scheduler that a fiber should have a preference on where it runs? Or is that something the OS itself handles somehow?

[ysbaddaden]

@Blacksmoke16 From what I read specifying a thread priority can hint the OS to schedule the thread on a big (efficient) or little (power) core. We can also set a thread affinity to a given core, but we must detect the core type beforehand.

[ysbaddaden]

Thinking again about names:

• ST: Mono? Single? Concurrent (to oppose on Parallel)?
• MT: Parallel is still the best for MT 👍
• I prefer Isolated over Exclusive after all (the fiber gets isolated), but no strong opinion;

I'm also thinking about simple constructors:

io_workers = ExecutionContext.concurrent
cpu_workers = ExecutionContext.parallel(size: 8)
ui = ExecutionContext.isolate { UI.main_loop }

Yet, I'm still struggling for a nice name to the single threaded context 😞

[straight-shoota]

SingleThreaded maybe?

I'm not sure we need such convenience constructors. This isn't essential anyway and we can figure it out later.

[ysbaddaden]

@straight-shoota yes, the convenience fonctions aren't needed, but while ExecutionContext::Parallel and ExecutionContext::Isolated feel nice, ExecutionContext::SingleThreaded doesn't have the same catchy feeling.

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/how-to-build-for-release/6808/5

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/timeline-for-multithreading-support/3604/21

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/upcoming-release-1-14-0/7199/1

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/new-event-loop-unix-call-for-reviews-tests/7207/6

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/charting-the-route-to-multi-threading-support/7320/1

[Qard]

Just wanted to share some thoughts/ideas I'm using in my own Fiber-based multi-threaded language.

There are two distinct access patterns that can be isolated into different types of Fiber construction:

• In the typical case it is desirable that a Fiber takes access to variables present in the lexical scope in which it is constructed. As such, distributing to another thread creates synchronization challenges which make doing so undesirable.
• In some cases you have a singular "task" you want distributed which it would be reasonable that the processor takes ownership of that in the sense that a language like Rust does.

How I chose to handle this is with two distinct Fiber types: a local Fiber which can access any in-scope data but can only execute on the thread in which it was created, and an ownership-claiming Fiber which specifically avoids running on the thread in which it was created, taking ownership of a passed in value and bringing that over with it to whatever thread it gets dispatched to.

On top of that I have a work-stealing mechanism which can pull a Fiber back to its origin thread when it's not busy, but the Fiber would otherwise be guaranteed to run on a different thread.

This design focuses very strongly on maintaining locality to achieve the best performance and to avoid synchronization complexity as much as possible while making distribution over threads certain when loop utilization would benefit from it.

In a typical app server you're going to see http requests come in which are largely isolated from each other and can be immediately distributed to other threads, but then the processing within those tends to need to share more with each other, and what you need within the scope of parts of that request is more likely just concurrency and not actually parallelism. If you need parallelism you can reach for that tool, but if a more restrictive tool is provided to ensure actual isolation of the work for that Fiber then it can be much more effectively distributed.

Another thing you see often with http requests is that most object references within a request are to objects allocated in that request, but there are a small number of exceptions which typically fit neatly into a shared interface category such as a database connection (or pool) or an app server exposing utilities to each route. For these types I opted to build actors into my language and have any interaction with these objects go through a shim copied into the ownership-taking fibers which dispatch these interactions across threads through Fiber yields as the "lock" mechanism. A task simply gets passed over to the thread owning the object to do that interaction and the result of that interaction resolves the yield on the other end.

In my language I went for an explicit form of this actor type specialization, but you could also go for explicit construction of a proxy type when passing to an ownership-taking Fiber.

In any case, what I wanted to convey is that in my experience there are two distinct ways in which people want to distribute work with Fibers and isolating those two behaviours can actually have a lot of benefits. 🙂

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/upcoming-release-1-15-0/7537/1