Add a version of Miou which is fully compatible with Picos

[dinosaure]

/cc @polytypic /cc @ada2k

This is a first attempt about a compatibility layer between Miou and picos. Actually, Miou uses only Picos.Trigger.Await, so a library which wants to work on Miou and another scheduler should just use Picos.Trigger and Picos.Computation: the rest is not handled by Miou¹.

I also think it's a drawback that Picos defines its own Picos_exn_bt.t type when the latter can be a simple tuple (and thus become a simple alias rather than a real type defined by Picos).

Once again, it's an experiment at this stage, which implies one thing above all: that @polytypic will henceforth be careful about API changes that may break code (welcome to the wonderful world of maintenance 😉).

Footnotes

1. This brings to the surface the question of the Current effect which, once again, seems to me better to give a single integer rather than the whole Fiber object. ↩

[c-cube]

Tuples are not great, but maybe we can make a exn-bt opam package with just the type definition (just like atomic, seq, result, etc.)? I'd have to lost the convenience of Exn_bt (which should have been in the stdlib ages ago anyway) :)

[ada2k]

What is the behaviour when a Picos function spawns a child? I've had libraries written for picos using Picos.Fiber.Spawn throw Miou.Still_has_children running under miou-with-picos.

[dinosaure]

What is the behaviour when a Picos function spawns a child? I've had libraries written for picos using Picos.Fiber.Spawn throw Miou.Still_has_children running under miou-with-picos.

Miou has a structured concurrency design. This means that a subtask launched within a task must be completed before its parent task is finished (otherwise, Still_has_childrenis raised by Miou). If you launch a task with picos, a child is added to the current task and will be cleaned when this subtask ends (with an error or not). You'd need to take a closer look at your code to see where the error is. But you can't “forget” children.

[polytypic]

What is the behaviour when a Picos function spawns a child? I've had libraries written for picos using Picos.Fiber.Spawn throw Miou.Still_has_children running under miou-with-picos.

Miou has a structured concurrency design. This means that a subtask launched within a task must be completed before its parent task is finished (otherwise, Still_has_childrenis raised by Miou). If you launch a task with picos, a child is added to the current task and will be cleaned when this subtask ends (with an error or not). You'd need to take a closer look at your code to see where the error is. But you can't “forget” children.

Unfortunately that does not match the Picos specification.

The Picos interface specifically and intentionally does not impose any particular structuring constraints on fibers spawned through it. The goal is to allow libraries using only the Picos interface to implement whatever structuring constraints they like.

As an example,the Picos_structured library implements a Bundle mechanism for structured concurrency. The Bundle.join_after (fun bundle -> (* scope *)) function guarantees that any fibers forked in to the bundle have terminated as the "scope" ends and before join_after returns.

Picos_structured also provides a rather small and simple Finally module that provides helpers for avoiding resource leaks in a very familiar manner (think: a refined version of Fun.protect for resource management). It also provides a way to explicitly move resources from one fiber to another. (The Finally module can be used with any scheduler and with any fiber structuring approach — it does not require using the Bundle module, for example.)

Like I mentioned, the goal for the Picos interface is to allow (essentially) any structuring constraints to be implemented as libraries that only depend on the Picos interface. This also goes for resource management.

Miou uses a different kind of Ownership mechanism (i.e. different compared to Picos_structured.Finally or Fun.protect) for managing resources — the idea of which it copied from the (currently unpublished) Oslo library. I mentioned on the Picos Discord some time ago

Couple of things I've also been thinking about lately is whether Fiber.t should also provide a way to register at_exit operations and whether there should be an explicit way for a scheduler to "dispose" a Fiber.t or mark a Fiber.t as having been disposed (possibly just a single bit). These can be partially done on top of the Picos interface already, but it might make sense to support these directly in the core interface.

and one, but not the only, motivation for this is to allow resource management similar to provided by Miou (and Oslo) to be implemented.

There is now a PR towards this. It aims to redesign the spawn interface of Picos and add new capabilities to the FLS to allow various things to be implemented on top of the Picos interface that were not previously implementable in a modular fashion. (Specifically, if you wanted some form of initialization and finalization, then you could, of course, write it on top of spawn, but then anything needing those would need to use a specific layer on top of spawn (i.e. would depend on a custom spawn that supports initialization and finalization) and would not be interoperable.)

At the time of writing this the PR is still a draft and work-in-progress — there are a number of design details that still need more thinking.

However, the PR also includes an example "Meow" library that provides an Ownership module and a Promise module that impose similar constraints as Miou for resources and structured concurrency. Please note that the "Meow" library, at this point, is a draft example and is not supposed to provide an absolutely exact copy of the semantics of Miou.

Again, the idea here is to allow these kind of behaviors to be implemented as modular libraries that depend only on the Picos interface. This way one can use those libraries with any schedulers as well as with any other libraries implemented in terms of just the Picos interface. This way all the schedulers and libraries can be made interoperable and modular. (It does require conscious effort and doesn't happen entirely automatically, because the Picos interface does not and cannot prevent you from building modules on top of it that depend on each others' implementation details.). In this case, for example, it would be entirely possible to use the Meow.Ownership module with other fiber structuring approaches (i.e. not just the Meow.Promise module) and, of course, any compatible scheduler.

[dinosaure]

I've just proposed a final patch in which a few tests taken from Picos (as far as Picos_structured is concerned) have been taken over and are working. The tests that have been eliminated mainly concern those that use Cancel_after, which is not supported.

I'm not very happy with this latest patch, which requires the use of an Obj.magic in certain places, as Fiber.t loses type information about what Computation.t should return - as Miou doesn't use Fiber.set_computation, we assume that the type couldn't change.

I'd like to re-introduce that this PR remains an experiment. It's normal that the result obtained may not correspond to what we theoretically expect: in other words, even if this PR lays the foundations for possible compatibility with Picos, I can't guarantee total compatibility with everything Picos can offer (and that's not my objective).

To clarify my position:

• picos.sync remains for me the essential to support
• picos.structured is also interesting but requires elements I can't support (especially Cancel_after)

So, venturing outside this framework is bound to come as a surprise when it comes to Miou's behavior and what one might imagine with Picos (considering its examples).

Above all, this PR should be seen as an effort on my part to create bridges, but also a concretization of these bridges with Picos (to get out of theory) with the emergence of “details” which, as far as I'm concerned, don't satisfy me at this stage.

From these details, we can concretely discuss with @polytypic a possible direction about Picos and/or Miou that can satisfy us all. The idea is to consolidate these bridges: but a bridge has two parts that must come together - this PR is the culmination of only one of them 😉.

[ada2k]

So, venturing outside this framework is bound to come as a surprise when it comes to Miou's behavior and what one might imagine with Picos (considering its examples).

Apologies if you caught a near incoherent complaint from me due to this - said code I was having trouble with actually was unstructured due to an incorrectly implemented per-domain loop system, but these patterns are harder to identify when they are being enforced at a different level to the code you are actually writing and do not appear in picos.fifos backed testcases.

If Picos provided these constraints, would it be expected for implementations like Miou to refuse picos code that is not explicitly written with Meow? (great for safety, but imposes the slight cost that libraries will have to target specific schedulers)