Enhanced new build_many to use on all platforms

[embray]

In #27490 I hacked together a replacement for the sage_setup.docbuild.build_many function, which implements (blocking) parallel map() of sorts, which solved some problems with using multiprocessing.Pool.map that stems from its use of threads to fork new processes.

That solved a problem that was specific to older versions of Cygwin. However, there is a similar problem, which affects all platforms, with PARI built with multi-threading support: #26608.

Although the PARI problem begs a more complete solution, at least for the docbuild we can get around it by using the build_many from #27490.

This ticket makes some fixes and enhancements to build_many, so that it can also return a result from the function it runs. For the docbuild this feature is not strictly needed, except for the fact that it can also be used (as in Pool.map) to raise any exception that occurs in one of the worker processes. Thus, it's closer in functionality, at least for the purposes of the docbuild, to Pool.map.

CC: @antonio-rojas @jdemeyer @kiwifb @dimpase @saraedum

Component: build

Keywords: docbuild parallel pari

Author: Erik Bray

Branch/Commit: e8d26b6

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/28356

[slel]

comment:2

Branch does not seem to merge.

[embray]

comment:3

Well it used to until a day ago...

[dimpase]

Changed branch from u/embray/docbuild/build-many-2 to public/docbuild/build-many-2

[dimpase]

comment:4

rebased branch

---

New commits:

65c7563	Enhance the new build_many to actually return results (including exceptions).
e8d26b6	Use new build_many on all platforms.

[dimpase]

Changed commit from bd92f13 to e8d26b6

[embray]

comment:5

Anyone want to go ahead and try using this?

[antonio-rojas]

comment:6

Built docs on Arch against system packages including threaded pari, can confirm it no longer segfaults.

[dimpase]

comment:7

looks good to me.

[dimpase]

Reviewer: Dima Pasechnik

[vbraun]

Changed branch from public/docbuild/build-many-2 to e8d26b6