Document build problem on CentOS 7

[kaigai]

When I tried to build the documentation on CentOS 7, using the packaged version of python-sphinx, its toolchain didn't work well because sphinx-build distributed with RPM package was too old. (python-sphinx-1.1.3-11.el7.noarch.rpm does not support -M option.)
It is helpful to describe the minimum required version.

[kaigai@namazu docs]$ make html
Sphinx v1.1.3
Usage: /usr/bin/sphinx-build [options] sourcedir outdir [filenames...]
Options: -b <builder> -- builder to use; default is html
         -a        -- write all files; default is to only write new and changed files
         -E        -- don't use a saved environment, always read all files
         -t <tag>  -- include "only" blocks with <tag>
         -d <path> -- path for the cached environment and doctree files
                      (default: outdir/.doctrees)
         -c <path> -- path where configuration file (conf.py) is located
                      (default: same as sourcedir)
         -C        -- use no config file at all, only -D options
         -D <setting=value> -- override a setting in configuration
         -A <name=value>    -- pass a value into the templates, for HTML builder
         -n        -- nit-picky mode, warn about all missing references
         -N        -- do not do colored output
         -q        -- no output on stdout, just warnings on stderr
         -Q        -- no output at all, not even warnings
         -w <file> -- write warnings (and errors) to given file
         -W        -- turn warnings into errors
         -P        -- run Pdb on exception
Modi:
* without -a and without filenames, write new and changed files.
* with -a, write all files.
* with filenames, write these.
make: *** [html] Error 1

Documentation could be built with the latest version download and overwritten with pip command, however, it was little bit inconvenient for CentOS/RHEL environment.

Thanks,

[kaigai]

Documentation could be built with the latest version download and overwritten with pip command,

No, it is still failed to build the document.

The following error messages are shown. I'm not sure what package provides /usr/lib/accelerate_radixsort.so, although message said it is just warnings.

[kaigai@namazu docs]$ make html
sphinx-build -M html source build
Running Sphinx v1.5.5
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] api
/home/kaigai/repo/pygdf/docs/source/api.rst:10: WARNING: autodoc: failed to import class u'DataFrame' from module u'pygdf.dataframe'; the following exception was raised:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/home/kaigai/repo/pygdf/pygdf/dataframe.py", line 9, in <module>
    from . import cudautils, utils
  File "/home/kaigai/repo/pygdf/pygdf/cudautils.py", line 8, in <module>
    from .sorting import RadixSort
  File "/home/kaigai/repo/pygdf/pygdf/sorting/__init__.py", line 1, in <module>
    from .radixsort import RadixSort
  File "/home/kaigai/repo/pygdf/pygdf/sorting/radixsort.py", line 38, in <module>
    lib = load_lib('radixsort')
  File "/home/kaigai/repo/pygdf/pygdf/sorting/common.py", line 26, in load_lib
    return ctypes.CDLL(libpath)
  File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/accelerate_radixsort.so: cannot open shared object file: No such file or directory
/home/kaigai/repo/pygdf/docs/source/api.rst:17: WARNING: autodoc: failed to import class u'Series' from module u'pygdf.dataframe'; the following exception was raised:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/home/kaigai/repo/pygdf/pygdf/dataframe.py", line 9, in <module>
    from . import cudautils, utils
  File "/home/kaigai/repo/pygdf/pygdf/cudautils.py", line 8, in <module>
    from .sorting import RadixSort
  File "/home/kaigai/repo/pygdf/pygdf/sorting/__init__.py", line 1, in <module>
    from .radixsort import RadixSort
  File "/home/kaigai/repo/pygdf/pygdf/sorting/radixsort.py", line 38, in <module>
    lib = load_lib('radixsort')
  File "/home/kaigai/repo/pygdf/pygdf/sorting/common.py", line 26, in load_lib
    return ctypes.CDLL(libpath)
  File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/accelerate_radixsort.so: cannot open shared object file: No such file or directory
/home/kaigai/repo/pygdf/docs/source/api.rst:26: WARNING: autodoc: failed to import class u'GpuArrowReader' from module u'pygdf.gpuarrow'; the following exception was raised:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/home/kaigai/repo/pygdf/pygdf/gpuarrow.py", line 13, in <module>
    from .dataframe import Series
  File "/home/kaigai/repo/pygdf/pygdf/dataframe.py", line 9, in <module>
    from . import cudautils, utils
  File "/home/kaigai/repo/pygdf/pygdf/cudautils.py", line 8, in <module>
    from .sorting import RadixSort
  File "/home/kaigai/repo/pygdf/pygdf/sorting/__init__.py", line 1, in <module>
    from .radixsort import RadixSort
  File "/home/kaigai/repo/pygdf/pygdf/sorting/radixsort.py", line 38, in <module>
    lib = load_lib('radixsort')
  File "/home/kaigai/repo/pygdf/pygdf/sorting/common.py", line 26, in load_lib
    return ctypes.CDLL(libpath)
  File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/accelerate_radixsort.so: cannot open shared object file: No such file or directory
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
generating indices... genindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 3 warnings.

Build finished. The HTML pages are in build/html.

build/html/*.html were generated, however, APIs section are empty...

Any idea?

[kaigai]

Does somebody check the issue?

[sklam]

Sorry for the late reply. To build the docs, it actually requires all runtime dependency to be present. There are dependencies that are only available through currently. See https://github.com/gpuopenanalytics/pygdf/blob/master/SETUP.md#conda-environments for instruction to setup the testing environment.

We should be able to avoid the need of runtime dependency by moving the import statements around.

[sklam]

Btw, we are planning to put the docs online soon. We are likely to publish it on readthedocs.io.

[kaigai]

Thanks for your help, however, /usr/lib/accelerate_radixsort.so was not still resolved in my centos7 environment.
According to the error message, sphinx script gets filed to open the shared library above, then it skips some files to be imported. Likely, it is the reason why the generated documents have empty chapters.
Which package provides /usr/lib/accelerate_radixsort.so?

[kaigai@namazu docs]$ source activate pycudf_testing_py35
(pycudf_testing_py35) [kaigai@namazu docs]$
(pycudf_testing_py35) [kaigai@namazu docs]$
(pycudf_testing_py35) [kaigai@namazu docs]$ make html
sphinx-build -M html source build
Running Sphinx v1.5.5
loading pickled environment... done
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 0 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] api
/home/kaigai/repo/pygdf/docs/source/api.rst:10: WARNING: autodoc: failed to import class u'DataFrame' from module u'pygdf.dataframe'; the following exception was raised:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/site-packages/sphinx/ext/autodoc.py", line 551, in import_object
    __import__(self.modname)
  File "/home/kaigai/repo/pygdf/pygdf/dataframe.py", line 9, in <module>
    from . import cudautils, utils
  File "/home/kaigai/repo/pygdf/pygdf/cudautils.py", line 8, in <module>
    from .sorting import RadixSort
  File "/home/kaigai/repo/pygdf/pygdf/sorting/__init__.py", line 1, in <module>
    from .radixsort import RadixSort
  File "/home/kaigai/repo/pygdf/pygdf/sorting/radixsort.py", line 38, in <module>
    lib = load_lib('radixsort')
  File "/home/kaigai/repo/pygdf/pygdf/sorting/common.py", line 26, in load_lib
    return ctypes.CDLL(libpath)
  File "/usr/lib64/python2.7/ctypes/__init__.py", line 360, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/accelerate_radixsort.so: cannot open shared object file: No such file or directory
      :
  <snip>
      :
OSError: /usr/lib/accelerate_radixsort.so: cannot open shared object file: No such file or directory
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
generating indices... genindex
writing additional pages... search
copying static files... done
copying extra files... done
dumping search index in English (code: en) ... done
dumping object inventory... done
build succeeded, 3 warnings.

Build finished. The HTML pages are in build/html.
(pycudf_testing_py35) [kaigai@namazu docs]$

[kaigai]

Btw, we are planning to put the docs online soon. We are likely to publish it on readthedocs.io.

+1
It seems to me better.

In case of PostgreSQL community (where I'm usually working on), developer writes up and modifies documents in SGML, and run document build chain,
https://github.com/postgres/postgres/tree/master/doc/src
however, people usually references static documents already built on web.
https://www.postgresql.org/docs/current/static/index.html

Best regards,

[sklam]

online docs is now available at http://pygdf.readthedocs.io/
=)