Future Development Crew / Mirai

[shikokuchuo]

Not to propose a new direction in development, but this does get me thinking big-picture about the mirai/crew integration. mirai has a dispatcher as a separate process which frees the user session. But in crew, because of auto-scaling and transient workers, the main R process needs to be an event loop that continuously calls push(), pop(), or wait() (similar to Gabor's task queue idea for callr). The crew event loop is even more important now that there is throttling. All this makes me wonder if crew itself could manually run an iteration of dispatch each time e.g. pop() is called, rather than overburdening the mirai dispatcher. It might not be feasible due to the scope and intent of mirai, but I keep wondering about this.

Originally posted by @wlandau in wlandau/crew#76 (comment)

[shikokuchuo]

On this I had an idea. I am prepared to provide access to dispatcher through a language interface.

By that I mean an argument at dispatcher() that takes a language object, or list of language objects, that can be evaluated in the context of the function itself at appropriate points in dispatcher's event loop.

As you know, the dispatcher event loop is hyper-efficient as it really is event-driven rather than polling. This means it is simply waiting most of the time. This is only possible due to its use of synchronisation primitives that nanonext makes available from NNG.

In this way, the additional scaling and launching logic can be retained within crew, and mirai can stick with its minimalist design purpose.

I would be happy to facilitate this if you find this appealing. I use this type of interface all the time in my machine learning work, so it's pretty much second nature to me. The ability for meta-programming / DSL is one of the core strengths of the R language and it would be rather cool to make use of it here.

@wlandau

[wlandau]

You would have the daemons outputs, but at the moment I don't think you would have the unresolved tasks - as they could be buffered at the C level (with NNG). I think this is one of those design points, where mirai takes a decentralised approach and does not keep track of state, whereas it is trivial for crew.

What kind of task-specific information is available? Presumably, I would see the number of tasks currently assigned to servers. Would I see the number of tasks that have not yet been dispatched?

Also when you say periodically, do you mean polling? I would be keen not to muddy the waters as dispatcher currently offers a clear demonstration of the benefits of the synchronisation primitives.

crew currently takes a polling-based approach, but maybe it wouldn't have to. What is the overall logical flow of the dispatcher event loop? I remember you said it spends most of its time sleeping, and I have noticed this makes crew super fast. Does it react to connections initiated by mirai::mirai() on the client? Is it involved in calls to unresolved() and $data?

I am starting to think a jointly-developed package implementing this more advanced dispatcher may be a more flexible solution.

Would it help if we set up some time to talk, maybe in late June?

[shikokuchuo]

What kind of task-specific information is available? Presumably, I would see the number of tasks currently assigned to servers. Would I see the number of tasks that have not yet been dispatched?

Currently dispatcher only materialises from buffer messages it can handle (i.e. same as number of servers). So there can be a lot of buffered incoming tasks that dispatcher does not know about yet.

crew currently takes a polling-based approach, but maybe it wouldn't have to. What is the overall logical flow of the dispatcher event loop? I remember you said it spends most of its time sleeping, and I have noticed this makes crew super fast. Does it react to connections initiated by mirai::mirai() on the client? Is it involved in calls to unresolved() and $data?

Yes, essentially it kicks into action whenever (i) there is a new server connection, (ii) a task is received from the client (subject to the limitation above so this is not the same as every time a task is sent) or (iii) a completed task is received from a server. Also when daemons() status is called or saisei(). The rest of the time it just waits. unresolved() and $data would not trigger dispatcher.

I am starting to think a jointly-developed package implementing this more advanced dispatcher may be a more flexible solution.

Would it help if we set up some time to talk, maybe in late June?

Sounds like a good idea!

[wlandau]

Yes, essentially it kicks into action whenever (i) there is a new server connection, (ii) a task is received from the client (subject to the limitation above so this is not the same as every time a task is sent) or (iii) a completed task is received from a server. Also when daemons() status is called or saisei(). The rest of the time it just waits. unresolved() and $data would not trigger dispatcher.

Thanks, this really helps me wrap my head around this.

Sounds like a good idea!

Awesome! Please let me know when you are back from vacation and we can coordinate something.

[shikokuchuo]

Given the questions, I wonder if promises may actually be what you're looking for as they are designed precisely for side-effecty stuff you presumably want to do to scale workers. Example:

library(promises)
library(mirai)
library(mirai.promises)
p1 <- as.promise(mirai({Sys.sleep(10); "complete"}))
p2 <- as.promise(mirai({Sys.sleep(10); "complete"}))
p1$then(cat)
p2$then(function() cat("\nhello"))

Or you may wish to call the side effect directly by modifying as.promise.mirai() in mirai.promises. You can see the logic of promises in that function in any case, you specify the eventual values for 'resolve' and 'reject', a check function that must not error or hang, and schedule the check function to poll using later::later().

[wlandau]

There are definitely appealing aspects of promises/later for auto-scaling. I thought about later a few weeks ago, but it gets tricky on the client, e.g. if the user's R session blocks for an hour and delays auto-scaling that would otherwise run with only a 0.25-second throttling delay. If I could inject code into the dispatcher event loop, which sleeps almost all of the time, I could definitely try later:

1. When the dispatcher kicks into action, create an auto-scaling promise which will asynchronously resolve 0.25-seconds from now.
2. If the dispatcher kicks into action again before the promise resolves, do nothing.
3. When 0.25 seconds are up, resolve the promise next time the dispatcher event loop is available.
4. The next time the dispatcher wakes up and does something, create a new auto-scaling promise that asynchronously resolves later.