Make Distributed.Worker threadsafe

[jonas-schulze]

I don't know how to test this more specifically, but the included tests failed reliably (on my machine) prior to the fix. If you have suggestions, I am happy to adjust/replace the tests.

The included commit by Kristoffer is an artifact I needed for development. I forgot to replace the calls to notify, which caused the build to hung up on creating some precompile statements (without printing any errors). Loading the changes to the stdlib using Revise didn't work on my machine, I don't know why. I can rebase and drop that commit if you like.

Fixes JuliaLang/Distributed.jl#73

[jonas-schulze]

Whoops, I lost a commit during rebase. While fixing that, I dropped the commit mentioned above. Sorry if I started two CI runs!

[vchuravy]

The test failure looks relevant:

  From worker 2:	  ArgumentError: Active workers with lazy=false. Cannot set lazy=true

[jonas-schulze]

Thanks for looking into the logs, @vchuravy! Should be fixed now. As the tests remove all the workers they added, I just moved them above the topology tests.

[jonas-schulze]

I have no idea why the new tests are that flaky.

[jonas-schulze]

The problem might be the shared in and outstreams:

• https://github.com/JuliaLang/julia/blob/master/stdlib/Distributed/src/managers.jl#L503
• https://github.com/JuliaLang/julia/blob/master/stdlib/Distributed/src/managers.jl#L511

Edit: I suspected an error in locking the socket during the handshake, but I can't tell for sure. On Linux it fails in libuv IOError: connect: address not available (EADDRNOTAVAIL), on BSD due to could not load symbol "jl_task_get_next": Shared object "libgmp" not found, required by "julia".

• tester_linux32, tester_win64 and tester_linuxaarch64: Distributed test suite passes
• tester_freebsd64 and tester_linux64 each have one test set fail in the following pattern:

Test Summary:                                            | Pass  Error  Total
RemoteChannel allows put!/take! from thread other than 1 |   56      2     58
...
    from thread *.1 to *.1                               |    4             4
    from thread *.2 to *.1                               |           1      1
    from thread *.1 to *.2                               |           1      1
    from thread *.2 to *.2                               |    4             4
...

[jonas-schulze]

I have no idea how to fix this. Even though the failure is clearly caused by the new tests, the reason seems unrelated to the changes proposed in this PR. Should I document the failed builds in a separate issue and move the tests to be a comment?

[vchuravy]

Can you rebase onto master one more time?

[jonas-schulze]

@vchuravy some tests are still failing, but it looks like for different reasons. This time they failed to connect to the workers (within the Distributed test suite).

[vchuravy]

Thank you Jonas!