Python-style imap()

[bjarthur]

Python provides a whole family of parallel map functions, one of which (imap in the multi-processing module) is non-blocking and returns an iterable list. It comes in very handy when the results wouldn't all fit in memory at once. Rather, each result is stored on the workers until de-referenced in the main thread. I provide this same functionality in julia with the new ipmap() function.

[JeffBezanson]

We want to have this, but it should not be implemented by copying and pasting the code for pmap. Maybe an option to pmap to make it return an array of RemoteRefs instead of results?

[tanmaykm]

Something similar was required to implement distributed DataFrames using Blocks.

The pmap implemented for Blocks (here: https://github.com/tanmaykm/Blocks.jl/blob/master/src/pmap.jl) takes an additional keyword fetch_results to achieve this by either calling remotecall_wait or remotecall_fetch. Its structure closely resembles Base.pmap and can be easily merged into base if acceptable.

[bjarthur]

I understand not wanting to duplicate code, but there is added benefit in ipmap not blocking the command line. That way subsequent code in the calling function can start processing the results as they are finished. @JeffBezanson and @tanmaykm, your solutions both block, no?

How about re-writing pmap to call ipmap? I have done this, and pushed. To show up here the pull request needs to be re-opened I think. If not, let me know if you want to submit a new pull request.

bjarthur@4bdec64

[amitmurthy]

Since julia multitasking is cooperative and not pre-emptive, it may not be a good idea to support processing results as they finish since

1. if the post-processing is compute heavy, it may block the pmap task from scheduling pending jobs from the input array
2. say, one of the results is on remote worker N, but if N is currently busy executing a different index from the input array, the fetch from N too will block rendering the design moot.

As Jeff suggested probably just supporting a kw argument specifying returning an array of RemoteRefs is a good compromise.