pmap with nprocs() == 1

[diegozea]

Should pmap ask for the number of process and fall into map when nprocs() == 1 ?

julia> const times = [0.001 for i=1:100];

julia> map(sleep, times);

julia> pmap(sleep, times);

julia> nprocs()
1

julia> @elapsed map(sleep, times)
0.209796101

julia> @elapsed map(sleep, times)
0.208756396

julia> @elapsed pmap(sleep, times)
0.212988051

julia> @elapsed pmap(sleep, times)
0.212867136

julia> @elapsed nprocs() == 1 ? map(sleep, times) : pmap(sleep, times)
0.209581963

julia> @elapsed nprocs() == 1 ? map(sleep, times) : pmap(sleep, times)
0.20959361

[kswietli]

To offer my view: It would be good for a different reason aside from performance. In my unrefined ways I often replace map with pmap back and forth. map for when I need a backtrace, pmap when I want things to go fast. It would be much easier for me to delete -p 4 from the command line instead of editing a file to change one line from pmap to map so that I get a traceback instead of a RemoteException.

[StefanKarpinski]

Or better still – RemoteExceptions could be made to display backtraces. That is more work, however.

[kswietli]

I'm having some major free time this summer, so I will add that to my list of Julia projects possibly to be done, right after RNG work, if someone from GSoC does not grab either before.

[kswietli]

Oh, I completely misunderstood what you wrote there. I did not realize you can get a backtrace out of a RemoteException and I somehow thought the functionality is not there yet. I was relying on the manual for multiprocessing which does not mention this at all, as far as I can tell. The stdlib documentation does, very briefly. The docs could use a bit of a hint here, I feel.

Well, that makes my job debugging my code much simpler! Many thanks for that.

Two suggestions, if I may: If we are going to drop to a regular map in some conditions, it might be an idea to also at that point print a warning of some kind. Something possibly has gone wrong at that point. Secondly, an option to automatically throw the contained exception from a RemoteException if one gets generated during a pmap? I imagine in many workloads when a pmap has gone bad, you probably just want the backtrace or the exception. pmap being designed for bigger workloads can likely afford to do a bit more logic within. Seems like the end user's code would be cleaner if we could pass another keyword to pmap as opposed to having them go and pick apart the return result of pmap for exceptions.

[amitmurthy]

RemoteExceptions do display backtraces.

Referencing #14843, @samoconnor is working on a pmap rewrite. A keyword to rethrow RemoteExceptions can be introduced. Maybe collect all individual remote exceptions into a CompositeException before the throw?

I am against calling map for nprocs() == 1 if the kind of exceptions thrown (RemoteException or the actual local exception) depends on nprocs()