Fix building html documentation on macOS

[slel]

Building html documentation fails on macOS for Sage 9.2.beta8.

Initial report:

• sage-release, 2020-08-13

Depends on #29552

CC: @jhpalmieri @slel @embray @kiwifb @kwankyu @tscrim @fchapoton @vbraun

Component: build

Author: Dima Pasechnik

Branch/Commit: 4b11a70

Reviewer: John Palmieri

Issue created by migration from https://trac.sagemath.org/ticket/30351

[slel]

comment:1

I'll try to bisect using the list of commits from 9.2.beta7 to 9.2.beta8:

• sage-release: Sage 9.2.beta8 released

At commit ed79ba3, make dochtml hangs. Now trying commit 9911c15.

[jhpalmieri]

comment:3

I think this is caused by #29766. I don't know why, but please try 9.2.beta7 first, then try 9.2.beta7 + #29766 and see if the problem arises.

[dimpase]

comment:4

Erik (in cc) was the last who changed parallel docbuild.

If noone understands the parallel docbuild system, it might be tempting to dump it, and replace by a straight sphinx build.

The current system looks like something inviting trouble, with explicit checks for OS flags, etc etc.

[slel]

comment:5

Not related to parallel docbuild a priori: I was always building with make -j1.

Homebrew-related. See below two trials on a MacBook Air (early 2014) with macOS 10.14.6 Mojave.

The error occurs when using Homebrew packages:

$ ORIGIN=https://github.com/sagemath/sage.git
$ git clone --single-branch --branch develop --tags $ORIGIN
$ cd sage
$ MAKE='make -j1'

$ source .homebrew-build-env

$ git checkout 9.2.beta7
$ git log --oneline -1
83caa4befa (HEAD, tag: 9.2.beta7) Updated SageMath version to 9.2.beta7

$ ./bootstrap
$ CC=clang CXX=clang++ OBJCC=clang OBJCXX=clang++ ./configure -q
$ make
--> OK

$ git checkout 3925b0f008
$ git log --oneline -2
3925b0f008 (HEAD) Trac #30230: Fix docstrings in sage/coding/linear_rank_metric.py
83caa4befa (HEAD, tag: 9.2.beta7) Updated SageMath version to 9.2.beta7

$ ./bootstrap && ./configure -q && make doc-clean && make
--> OK

$ git checkout 34e01d1ef4
$ git log --oneline -3
34e01d1ef4 (HEAD) Trac #29766: NumPy 1.19.1, SciPy 1.5.2, networkx 2.4, pybind11
3925b0f008 Trac #30230: Fix docstrings in sage/coding/linear_rank_metric.py
83caa4befa (tag: 9.2.beta7) Updated SageMath version to 9.2.beta7

$ ./bootstrap && ./configure -q && make doc-clean && make
--> hangs on "constructions"

It does not occur when using the least system things possible:

$ ORIGIN=https://github.com/sagemath/sage.git
$ git clone --single-branch --branch develop --tags $ORIGIN
$ cd sage
$ MAKE='make -j1'

$ git checkout 9.2.beta7
$ git log --oneline -1
83caa4befa (HEAD, tag: 9.2.beta7) Updated SageMath version to 9.2.beta7

$ ./bootstrap
$ CC=clang CXX=clang++ OBJCC=clang OBJCXX=clang++ \
  ./configure \
  --enable-openssl=yes \
  --with-python=3 \
  --with-system-arb=no \
  --with-system-gcc=no \
  --with-system-flint=no \
  --with-system-mpfr=no \
  --with-system-mpir=no \
  --with-mp=mpir \
  --with-system-ntl=no \
  --with-system-openblas=no \
  --with-blas=openblas \
  --with-system-boost_cropped=no \
  --with-system-brial=no \
  --with-system-m4ri=no \
  --with-system-libpng=no \
  --with-system-zlib=no \
  --with-system-bzip2=no \
  --with-system-cddlib=no \
  --with-system-cliquer=no \
  --with-system-curl=no \
  --with-system-eclib=no \
  --with-system-pari=no \
  --with-system-readline=no \
  --with-system-ncurses=no \
  --with-system-ecm=no \
  --with-system-fflas_ffpack=no \
  --with-system-givaro=no \
  --with-system-flintqs=no \
  --with-system-fplll=no \
  --with-system-freetype=no \
  --with-system-gf2x=no \
  --with-system-gfan=no \
  --with-system-glpk=no \
  --with-system-gsl=no \
  --with-system-iconv=no \
  --with-system-iml=no \
  --with-system-lcalc=no \
  --with-system-libatomic_ops=no \
  --with-system-libbraiding=no \
  --with-system-libffi=no \
  --with-system-libgd=no \
  --with-system-lrcalc=no \
  --with-system-m4rie=no \
  --with-system-mpc=no \
  --with-system-mpfi=no \
  --with-system-nauty=no \
  --with-system-palp=no \
  --with-system-pari_galdata=no \
  --with-system-pari_nftables=no \
  --with-system-pari_seadata_small=no \
  --with-system-patch=no \
  --with-system-pcre=no \
  --with-system-pkgconf=no \
  --with-system-planarity=no \
  --with-system-ppl=no \
  --with-system-python3=no \
  --with-system-sqlite=no \
  --with-system-xz=no \
  --with-system-r=no \
  --with-system-rw=no \
  --with-system-suitesparse=no \
  --with-system-symmetrica=no \
  --with-system-sympow=no \
  --with-system-tachyon=no \
  --with-system-yasm=no \
  --with-system-zeromq=no \
  -q
$ make openssl && make
--> OK

$ git checkout 3925b0f008
$ git log --oneline -2
3925b0f008 (HEAD) Trac #30230: Fix docstrings in sage/coding/linear_rank_metric.py
83caa4befa (HEAD, tag: 9.2.beta7) Updated SageMath version to 9.2.beta7

$ ./bootstrap && ./configure -q && make doc-clean && make
--> OK

$ git checkout 34e01d1ef4
$ git log --oneline -3
34e01d1ef4 (HEAD) Trac #29766: NumPy 1.19.1, SciPy 1.5.2, networkx 2.4, pybind11
3925b0f008 Trac #30230: Fix docstrings in sage/coding/linear_rank_metric.py
83caa4befa (tag: 9.2.beta7) Updated SageMath version to 9.2.beta7

$ ./bootstrap && ./configure -q && make doc-clean && make
--> OK

Next piece of fun could be: isolate which Homebrew package causes trouble.

[jhpalmieri]

comment:6

The problem goes away if I do

./configure --with-system-gsl=no --with-system-openblas=no

(I have Homebrew packages for these installed.)

Note that

./configure --with-system-openblas=no

triggers building both openblas and gsl. Note also that gsl fails to build with

./configure --with-system-gsl=no

The log file complains

configure: error: in `/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta8/local/var/tmp/sage/build/gsl-2.5/src':
configure: error: C compiler cannot create executables

So I don't know if it's Homebrew's openblas or gsl that's causing the problem.

---

Actually, I really don't understand the gsl issue. I started with make distclean and ./configure --with-system-gsl=no. Then if I do make, it builds about half a dozen packages (cliquer, m4ri, pari_galdata, pari_seadata_small, libatomic_ops) before starting the gsl build, which fails with the error above. The gsl config.log file says

configure:3482: checking whether the C compiler works
configure:3504: gcc -g -O2   -L/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta8/local/lib -Wl,-rpath,/Users/palmieri/Desktop/Sage_stuff/sage_builds/TESTING/sage-9.2.beta8/local/lib  conftest.c -lopenblas -lm >&5
ld: library not found for -lopenblas
clang: error: linker command failed with exit code 1 (use -v to see invocation)

The confusing part: if I instead do make distclean, ./configure --with-system-gsl=no, and then make gsl, it succeeds. If I build the half dozen packages by hand (make cliquer etc.), then make gsl succeeds. So I don't know how to debug this side issue. Does plain make do something extra that make PKG doesn't do?

[jhpalmieri]

comment:7

Given my problems with gsl, take this with a grain of salt, but with ./configure --with-system-gsl=no, if I then do make gsl followed by make, the process hangs while building the documentation. So the Homebrew package openblas seems to be the problem, since the documentation builds with ./configure --with-system-gsl=no --with-system-openblas=no.

Can anyone else verify?

[mkoeppe]

comment:8

At which point during the docbuild does it hang?

Does make doc-html-no-plot also hang?

[mkoeppe]

comment:9

I think I have reproduced the bug on my system.
Using

./sage -python -m pdb -m sage_setup.docbuild --verbose 3 --no-pdf-links all html

and then interrupting when it has stalled shows that it's in wait_for_one, waiting for a child to exit.

[mkoeppe]

comment:10

I'm hoping that people who know about sage_setup.docbuild can help here.

It hangs in sage_setup.docbuild.utils.build_many.

It needs to be investigated whether one of the worker processes hangs and therefore os.wait waits forever, or whether something else is the problem.

[mkoeppe]

comment:11

Replying to @mkoeppe:

It needs to be investigated whether one of the worker processes hangs and therefore os.wait waits forever

This seems to be the case. Easier to test by setting SAGE_NUM_THREADS=1. Killing the worker process (unique child) with kill -INT had no effect. Killing it with kill -TERM leads to:

  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage_setup/docbuild/utils.py", line 285, in build_many
    raise worker_exc
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage_setup/docbuild/utils.py", line 257, in build_many
    if reap_workers(waited_pid, waited_exitcode):
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage_setup/docbuild/utils.py", line 215, in reap_workers
    w = bring_out_yer_dead(w, task, exitcode)
  File "/Users/mkoeppe/s/sage/sage-rebasing/worktree-algebraic-2018-spring/local/lib/python3.7/site-packages/sage_setup/docbuild/utils.py", line 153, in bring_out_yer_dead
    "{}".format(task[1], w.exitcode))
sage_setup.docbuild.utils.WorkerDiedException: worker for ('en/constructions', 'html', {}) died with non-zero exit code -15
Uncaught exception. Entering post mortem debugging

[jhpalmieri]

comment:12

I don't have time over the next few days, but two potential parallel lines of investigation: why do Homebrew's openblas combined with #29766 cause this? And are there problems in the rst source for constructions and thematic_tutorials that, if fixed, could avoid this issue?

[mkoeppe]

comment:13

Some superficial debugging revealed the following: The worker process is stuck with all threads waiting in __psynch_cvwait (invoked from the depths of a BLAS call). Why this happens is unclear.

Some directions to explore:

• the ad-hoc multiprocessing code in sage_setup.docbuild might interfere with the pthreads implementation
• gsl confused about gslblas vs openblas

[mkoeppe]

comment:14

This is not good:

local/lib/libgiac.dylib:
        /Users/mkoeppe/s/sage/sage-rebasing/worktree-homebrew-py38/local/lib/libgiac.0.dylib (compatibility version 1.0.0, current version 1.0.0)
        /usr/local/opt/ntl/lib/libntl.43.dylib (compatibility version 44.0.0, current version 44.1.0)
        /Users/mkoeppe/s/sage/sage-rebasing/worktree-homebrew-py38/local/lib/libpari-gmp.dylib (compatibility version 2.11.0, current version 2.11.4)
        /usr/local/opt/gsl/lib/libgsl.25.dylib (compatibility version 26.0.0, current version 26.0.0)
        /usr/local/opt/gsl/lib/libgslcblas.0.dylib (compatibility version 1.0.0, current version 1.0.0)

[kiwifb]

comment:15

Yes, giac link to gslcblas if it can find it by default when gsl is enabled. Even if you provide a blas/lapack. A long time ago I rewrote that part of the configure.in in sage-on-gentoo to avoid any double linking of cblas.
https://github.com/cschwan/sage-on-gentoo/blob/master/sci-mathematics/giac/files/giac-1.5.0.87-gsl_lapack.patch

[slel]

Changed keywords from none to macos, openblas, glsblas, giac

[kiwifb]

Changed keywords from macos, openblas, glsblas, giac to none

[kiwifb]

comment:17

The patch didn't include anything about cblas because I don't think cblas is used in giac's code, it is only pulled by libgsl which need a cblas. If you link libgsl against a cblas it should all work whatever it is without providing it for linking to libgiac.

[mkoeppe]

comment:18

homebrew's libgsl is underlinked and can be linked against openblas.
But its gsl.pc uses gslcblas...:

prefix=/usr/local/Cellar/gsl/2.6
exec_prefix=/usr/local/Cellar/gsl/2.6
libdir=/usr/local/Cellar/gsl/2.6/lib
includedir=/usr/local/Cellar/gsl/2.6/include
GSL_CBLAS_LIB=-lgslcblas

Name: GSL
Description: GNU Scientific Library
Version: 2.6
Libs: -L/usr/local/Cellar/gsl/2.6/lib -lgsl ${GSL_CBLAS_LIB}  -lm 
Cflags: -I/usr/local/Cellar/gsl/2.6/include

[mkoeppe]

comment:19

Probably best to take care of this on the giac upgrade ticket #29552

[kiwifb]

comment:20

I never tried it, is it possible to override .pc defined variable by environment variables? I would guess not but that would be interesting.

[mkoeppe]

comment:21

We already take care of the gsl.pc file by installing a patched version in $SAGE_LOCAL/lib/pkgconfig.

So only libraries that do not use pkgconfig can still fall into the gslblas trap

[kiwifb]

comment:22

Just for the record

$ pkg-config --libs gsl
-lgsl -lcblas -lm
$ pkg-config --define-variable=GSL_CBLAS_LIB=-lfoo --libs gsl
-lgsl -lfoo -lm

[mkoeppe]

comment:23

Sure, but the problem is of course that the pkg-config invocation is usually in a package's configure script, so one cannot add the --define-variable without patching.

[kiwifb]

comment:24

Replying to @mkoeppe:

Sure, but the problem is of course that the pkg-config invocation is usually in a package's configure script, so one cannot add the --define-variable without patching.

Unfortunately you are right. It is even less useful than I thought it could be.

[dimpase]

comment:66

The question is whether allowing OMP_NUM_THREADS>1 after the build leaves Sage broken - I think tests in src/sage/parallel/ need to be improved to involve numpy calls (with OMP_NUM_THREADS>1) to test for the situation we have here.

[jhpalmieri]

comment:67

I think adding doctests would be good. For me, though, running doctests on the file src/doc/en/a_tour_of_sage/index.rst leads to a hang (as you say in comment:54), but running those tests one by one in the Sage console does not lead to a hang. Does this mean that Sage itself is not broken, only something in the doctesting framework?

[dimpase]

comment:68

The problem is that Sage (and Python - cf. attachment :-)) contains multiprocessing/parallel stuff which uses fork(), and thus does not play well with extension modules, e.g. numpy, which may use OpenMP (in particular an implementation of OpenMP that is not fork-safe, such as libgopenmp).

So in principle this means that Sage was already broken years ago :-)

I think that in addition to the patch at comment:65, some docs should be updated to reflect the issue.

[dimpase]

comment:69

I tried

--- a/src/bin/sage-runtests
+++ b/src/bin/sage-runtests
@@ -27,6 +27,9 @@ def _get_optional_defaults():
 
 
 if __name__ == "__main__":
+    import multiprocessing
+    multiprocessing.set_start_method('spawn')
+
     parser = optparse.OptionParser()
 
     def optional_argument(option, opt_str, value, parser, typ, default_arg):

instead of os.environ["OMP_NUM_THREADS"] = "1", but this immediately errors out with

$ make test
...
Traceback (most recent call last):
  File "/Users/dima/software/sagetrac-mirror/src/bin/sage-runtests", line 182, in <module>
    err = DC.run()
  File "/Users/dima/software/sagetrac-mirror/local/lib/python3.7/site-packages/sage/doctest/control.py", line 1230, in run
    self.run_doctests()
  File "/Users/dima/software/sagetrac-mirror/local/lib/python3.7/site-packages/sage/doctest/control.py", line 931, in run_doctests
    self.dispatcher.dispatch()
  File "/Users/dima/software/sagetrac-mirror/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2046, in dispatch
    self.parallel_dispatch()
  File "/Users/dima/software/sagetrac-mirror/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 1941, in parallel_dispatch
    w.start()  # This might take some time
  File "/Users/dima/software/sagetrac-mirror/local/lib/python3.7/site-packages/sage/doctest/forker.py", line 2213, in start
    super(DocTestWorker, self).start()
  File "/usr/local/Cellar/[email protected]/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/usr/local/Cellar/[email protected]/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/usr/local/Cellar/[email protected]/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/usr/local/Cellar/[email protected]/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/usr/local/Cellar/[email protected]/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/usr/local/Cellar/[email protected]/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/usr/local/Cellar/[email protected]/3.7.8_1/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'DocTestDispatcher.parallel_dispatch.<locals>.sel_exit'

This is due to spawn using a different way to copy local data into another thread - probably it can be fixed, but not so easy.

---

similar problem pops up trying spawn with the docbuilding.

[dimpase]

Changed work issues from ensure OMP_NUM_THREADS=1 to none

[dimpase]

Reviewer: John Palmieri

[dimpase]

Author: Dima Pasechnik

[dimpase]

Branch: public/ticket/30351public/ticket/30351

[dimpase]

Changed branch from public/ticket/30351public/ticket/30351 to public/ticket/30351

[dimpase]

Commit: f8dcff3

[dimpase]

New commits:

f8dcff3

on macOS set OMP_NUM_THREADS to 1 for docbuild and tests

[dimpase]

comment:72

to try this on home-brew, one needs to install a fresh openblas, which in turn needs #29456 (that's why I put the latter In deps, it's not yet in beta)

[dimpase]

Changed dependencies from #29552 to #29552, #29456

[dimpase]

comment:73

I've tested this on the macOS 10.15.6, with updated Homebrew with all the packages we can use in Sage, used (with #29552, #29456) - all tests pass.

[sagetrac-git]

Changed commit from f8dcff3 to 488bba4

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

488bba4

on macOS set OMP_NUM_THREADS to 1 for docbuild and tests

[dimpase]

comment:75

rebased over beta10

[dimpase]

Changed dependencies from #29552, #29456 to #29552

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

4b11a70

use the correct import location, not six.*

[sagetrac-git]

Changed commit from 488bba4 to 4b11a70

[jhpalmieri]

comment:77

This works for me (although I didn't use #29552).

[dcoudert]

comment:78

while running make ptest, I got the following time outs

sage -t --random-seed=0 src/sage/dynamics/arithmetic_dynamics/projective_ds.py  # Timed out
sage -t --random-seed=0 src/sage/manifolds/differentiable/tensorfield.py  # Timed out
sage -t --random-seed=0 src/sage/plot/plot.py  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/base.pyx  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/implicit_plot3d.py  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/parametric_plot3d.py  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/plot3d.py  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/shapes.pyx  # Timed out
sage -t --random-seed=0 src/sage/plot/plot3d/shapes2.py  # Timed out
sage -t --random-seed=0 src/sage/schemes/elliptic_curves/ell_number_field.py  # Timed out

but the tests pass independently, e.g.,

sapristi:sage dcoudert$ ./sage -tp src/sage/dynamics/arithmetic_dynamics/projective_ds.py
Running doctests with ID 2020-08-25-19-17-07-e834b51b.
Git branch: HEAD
Using --optional=benzene,bliss,build,dochtml,igraph,plantri,python_igraph,sage,tdlib
Doctesting 1 file using 8 threads.
sage -t --warn-long 138.0 --random-seed=0 src/sage/dynamics/arithmetic_dynamics/projective_ds.py
    [1238 tests, 71.66 s]
----------------------------------------------------------------------
All tests passed!
----------------------------------------------------------------------
Total time for all tests: 71.9 seconds
    cpu time: 59.7 seconds
    cumulative wall time: 71.7 seconds

So for me it's ok too. Thank you all !

[vbraun]

Changed branch from public/ticket/30351 to 4b11a70