Revert "Merge pull request #38405 from JuliaLang/vc/distributed_ts"

[vtjnash]

Also revert "fixup to pull request #38405 (#41641)"

Seems to be causing hanging in CI testing? @vchuravy

This reverts commit 5af1cf0.
This reverts commit 5a16805, reversing
changes made to 02807b2.

[JeffBezanson]

Do you have a link to an example failure?

[jpsamaroo]

As an alternative to reverting this kind of PR for the fourth? time, can we try to just disable the failing test, if it's the one introduced by #38405? I think reverting it is equivalent to saying that #38405 is introducing a bug, but realistically it's probably just fixing one bug, and accidentally exposing another which already exists.

[vchuravy]

@JeffBezanson I think https://build.julialang.org/#/builders/69/builds/1798/steps/5/logs/stdio is the kind of issue that Jameson is concerned around. We have been observing a general increase in flakiness in the tests, either hanging or otherwise misbehaving often around Distributed.

Then there is also https://build.julialang.org/#/builders/9/builds/1640 and another example https://build.julialang.org/#/builders/78/builds/1578

can we try to just disable the failing test,

If it were just a failing test...

[DilumAluthge]

Did the rr trace yield anything useful?

[vtjnash]

Looking closer, the last log Valentin attached clearly shows this PR needs to be reverted and worked on more, as it is failing due the any_gc_flag lock in the new code on line 260:

  5 const any_gc_flag = Threads.Condition()
  4 function start_gc_msgs_task()
  3     errormonitor(
  2         Threads.@spawn begin
  1             while true
260                 lock(any_gc_flag) do
  1                     wait(any_gc_flag)
  2                     flush_gc_msgs() # handles throws internally
  3                 end
  4             end
  5         end
  6     )
  7 end