Add Portal.run_func(): run local funcs remotely
#105
Conversation
This is to address the request from a couple people for the ability to not have to specify the explicit module path for a callable to be invoked in a remote actor when the desired function is defined locally. This makes the API to run a local func look more like `trio.Nurser.start_soon(my_func, *args)` which is simpler and much more "shorthand". In this case we simply introspect the function/callable object and send that info to the remote actor to be *looked up locally* and scheduled. Resolves #69
There was a problem hiding this comment.
I understand your objection, and this is relatively sane approach to what you're trying to do here. My thinking with this suggestion was that I was wanting to consider tractor to be "trio across machines", which is a really, really cool idea. I think that the names are, essentially, how we would expect to be doing things under the hood. It's how Celery works, too. My question though, is if it really is a public interface, or if it's an implementation detail.
If it's a public interface, then we should expect to see users being very careful about compatibility between different versions of code on different machines. They should want to, and be expected to, consider the sending side and the receiving side separately. The calling side might have no understanding of what code might actually run for a particular message. If a unified vision was desired, it really would have to be built on top of tractor.
That might not the mode that tractor is looking to primarily fill, but that is what I was hoping tractor would be. I think that, internally, it's important to have the mode you're talking about, to make the interface explicit. I just also think that, ideally in my view, it should be a lower-level API for advance uses, rather than the Right Way to use tractor for the general case.
I envision that this layer on top of the tractor plumbing would be able to take a function, convert it to some serialized form, send it over the wire to the remote host, de-serialize it, and run it using the tractor plumbing. How exactly the serialization happens, or the deserialization happens, is an implementation detail that I expect many users need not care about, unless they start dipping into advanced uses.
In some cases it could be looking up a function's module name and path, and sending that. In other cases I could even imagine it serializing the function itself to replay on the remote host, using something like cloudpickle (though I can also imagine impassible challenges attempting to actually do that). I can imagine checks on deserialize that ensure that the destination is running the same code as the sender serialized from.
If that interface is not the "default" interface of tractor, then that would seem to me to be an indication that, if such a user-friendly layer were desired, it really should be part of a different package that wraps tractor, instead of a part of tractor itself. Does that seem correct to you?
If tractor is intending to be that, then I would expect that the registration of the functions to names in a registry should be very explicit, but that does not seem to be the case from the readme, although there's definitely room for me to still not be understanding it correctly.
Actually, all of this talking is making me feel like I just haven't spent enough time actually using tractor, to get a good feel for where you're trying to position it. I'm looking for that "trio across processes". You may be targeting a more low-level primitive, or a model that doesn't cleanly fit into what I'm looking for. So take all of my talk above for what they're worth: the idle ramblings of someone with his own ideas, that may not really reflect the ideas you wish to solve.
|
@ryanhiebert really appreciate this feedback! Maybe we can continue this discussion in #69? This is something that I'll follow up in the original issue referencing some of what you've wrote here. |
This is to address the request from a couple people (@parity3, @ryanhiebert) for the ability to
not have to specify the explicit module path for a callable to be
invoked in a remote actor when the desired function is defined locally.
This makes the API to run a local func look more like
trio.Nurser.start_soon(my_func, *args)which is simpler and much more"shorthand". In this case we simply introspect the function/callable
object and send that info to the remote actor to be looked up locally
and scheduled.
I'm not sold on making
Portal.run()take function objects since it simply won't easily translate to multi-host setups where remote code is not the same as local code. If this kind of API was sane you'd likely have seen it show up in widespread app layer IPC protocols like HTTP. It makes more sense to me to have a url/explicit module lookup system to determine task endpoints/entrypoints. With this premise in mind this PR proposes thatPortal.run()stay as is (maybe later accepting a URL-like syntax) andPortal.run_func()be used to easily run a locally defined function in a remote actor.Resolves #69
Note this is partially to open discussion since I'm open to other ideas of course.
This also still needs tests!