Create, publish public API documentation on readthedocs.io

.readthedocs.yml

@@ -0,0 +1,17 @@
+# .readthedocs.yml
+# Read the Docs configuration file
+# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
+
+# Required
+version: 2
+
+python:
+  version: 3.7
+  install:
+    - method: pip
+      path: .
+      extra_requirements:
+        - docs
+
+sphinx:
+  fail_on_warning: true

README.md

@@ -2,6 +2,8 @@
  src="https://raw.githubusercontent.com/bluelabsio/records-mover/master/docs/records-mover-horizontal.png"
  alt="Records Mover">
 
+[![Documentation Status](https://readthedocs.org/projects/records-mover/badge/?version=publish_docs)](https://records-mover.readthedocs.io/en/publish_docs/?badge=latest)
+
 Records mover is a command-line tool and Python library you can
 use to move relational data from one place to another.
 
@@ -92,8 +94,9 @@ results = move(source, target)
 ```
 
 When moving data, the sources supported can be found
-[here](./records_mover/records/sources/factory.py), and the
-targets supported can be found [here](./records_mover/records/targets/factory.py).
+[here](https://records-mover.readthedocs.io/en/publish_docs/records_mover.records.sources.html),
+and the targets supported can be found
+[here](https://records-mover.readthedocs.io/en/publish_docs/records_mover.records.targets.html).
 
 ## Advanced Python library use
 
@@ -138,3 +141,15 @@ target = targets.table(schema_name='myschema',
                        db_engine=db_engine)
 results = move(source, target)
 ```
+
+## Python library API documentation
+
+You can can find more API documentation
+[here](https://records-mover.readthedocs.io/en/publish_docs/index.html).
+In particular, note:
+
+* [Session() constructor](https://records-mover.readthedocs.io/en/publish_docs/records_mover.html#records_mover.Session.__init__)
+* [sources factory methods](https://records-mover.readthedocs.io/en/publish_docs/records_mover.records.sources.html)
+* [targets factory methods](https://records-mover.readthedocs.io/en/publish_docs/records_mover.records.targets.html)
+* [move() method](https://records-mover.readthedocs.io/en/publish_docs/records_mover.records.html#records_mover.records.move)
+* [BaseRecordsFormat](https://records-mover.readthedocs.io/en/publish_docs/records_mover.records.html#records_mover.records.base_records_format.BaseRecordsFormat)

deps.sh

@@ -20,4 +20,4 @@ pip3 install --upgrade pip
 # It's nice to unit test, integration test, and run the CLI in
 # a development pyenv.
 #
-pip3 install -r requirements.txt -e '.[unittest,itest,cli,typecheck]'
+pip3 install -r requirements.txt -e '.[unittest,itest,cli,typecheck,docs]'

docs/MAINT.md

@@ -19,8 +19,8 @@ Packages inside include:
   things that haven't grown enough mass to be exported into their own
   package.
 
-Things either labeled private with a prefix of `_` aren't stable
-interfaces - they can change rapidly.
+Things either labeled private, not in `__all__`, undocumented or with
+a prefix of `_` aren't stable interfaces - they can change rapidly.
 
 If you need access to another function/class, please submit an issue
 or a PR make it public.  That PR is a good opportunity to talk about
@@ -75,6 +75,20 @@ To run the same suite with mover itself in a Docker image:
 Try expanding your Docker for Mac memory size to 8G.  Vertica is
 memory intensive, even under Docker.
 
+### Documentation
+
+API reference documentation is pushed up to
+[readthedocs](https://records-mover.readthedocs.io/en/publish_docs/) by a
+GitHub webhook.
+
+To create docs, run this from the `docs/` directory:
+
+* `make html`
+
+To view docs:
+
+* `open build/html/index.html`
+
 ### Manual integration testing
 
 There's also a manual records schema JSON functionality

docs/Makefile

@@ -0,0 +1,23 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?= -W
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+apidocs:
+	@sphinx-apidoc -o ./source ../records_mover
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/source/conf.py

@@ -0,0 +1,75 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+sys.path.insert(0, os.path.abspath('../..'))
+import records_mover  # noqa
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'records-mover'
+copyright = '2020 BlueLabs Analytics, Inc.'
+author = 'Vince Broz'
+
+# The full version, including alpha/beta/rc tags
+release = records_mover.version.__version__
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = []
+
+#
+# This value selects if automatically documented members are sorted
+# alphabetical (value 'alphabetical'), by member type (value
+# 'groupwise') or by source order (value 'bysource'). The default is
+# alphabetical.
+#
+# records mover picks 'groupwise' so that enums appear in the order
+# listed in the source code, which can reflect a natural order.
+#
+autodoc_member_order = 'bysource'
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'sphinx_rtd_theme'
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# -- Options for type generation ---------------------------------------------
+
+#
+# Document parameter types in description of parameter, not in
+# function signature part of documentation.
+#
+autodoc_typehints = 'description'

docs/source/index.rst

@@ -0,0 +1,21 @@
+.. records-mover documentation master file, created by
+   sphinx-quickstart on Fri Jun 26 11:46:58 2020.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to records-mover's documentation!
+=========================================
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   records_mover
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/source/records_mover.airflow.hooks.rst

@@ -0,0 +1,16 @@
+records\_mover.airflow.hooks package
+====================================
+
+Module contents
+---------------
+
+.. comment:
+   SqlAlchemyDbHook.validate_and_prepare_target_directory is something
+   I'd like to explore removing and promoting to something
+   non-Airflow-related, so let's not include it in the docs and leave
+   'undoc-members' out
+
+.. automodule:: records_mover.airflow.hooks
+   :members:
+   :show-inheritance:
+   :special-members: __init__

docs/source/records_mover.airflow.rst

@@ -0,0 +1,10 @@
+records\_mover.airflow package
+==============================
+
+Subpackages
+-----------
+
+.. toctree::
+   :maxdepth: 4
+
+   records_mover.airflow.hooks

docs/source/records_mover.records.rst

@@ -0,0 +1,82 @@
+records\_mover.records package
+==============================
+
+Subpackages
+-----------
+
+.. toctree::
+   :maxdepth: 4
+
+   records_mover.records.sources
+   records_mover.records.targets
+
+Submodules
+----------
+
+records\_mover.records.base\_records\_format module
+---------------------------------------------------
+
+.. autoclass:: records_mover.records.base_records_format.BaseRecordsFormat
+   :members:
+   :show-inheritance:
+   :special-members: __init__
+
+Module contents
+---------------
+
+.. comment:
+
+   I exclude PartialRecordsHints entirely below because Sphinx doesn't
+   yet support TypedDict types that have hyphens as part of the keys.
+   Example error on 'make html':
+
+   WARNING: invalid signature for autoinstanceattribute
+   ('records_mover.records::PartialRecordsHints.header-row')
+
+   I exclude RecordsSchema as we haven't defined yet exactly what
+   public API to export.
+
+   https://github.com/bluelabsio/records-mover/issues/97
+
+   I exclude RecordsFormat as it's generally an internal factory
+   method useful when reading undifferentiated JSON files - a general
+   user ought to be able to select their records format in advance and
+   use the correct subclass.
+
+   I exclude UntypedRecordsHints and RecordsFormatType as it hasn't
+   been used in any other part of the public interface.
+
+   I exclude DelimitedVariant as there's no way to specify a
+   docstring, so I add out-of-band documentation below.
+
+.. automodule:: records_mover.records
+   :members:
+   :undoc-members:
+   :show-inheritance:
+   :exclude-members: DelimitedRecordsFormat, ParquetRecordsFormat, ProcessingInstructions, PartialRecordsHints, RecordsSchema, RecordsFormat, UntypedRecordsHints, RecordsFormatType, DelimitedVariant
+
+   .. autoclass:: records_mover.records.DelimitedRecordsFormat
+      :undoc-members:
+      :show-inheritance:
+      :special-members: __init__
+
+   .. autoclass:: records_mover.records.ParquetRecordsFormat
+      :undoc-members:
+      :show-inheritance:
+      :special-members: __init__
+
+   .. autoclass:: records_mover.records.ProcessingInstructions
+      :undoc-members:
+      :show-inheritance:
+      :special-members: __init__
+
+   .. py:data:: DelimitedVariant
+      :module: records_mover.records
+
+      Valid string values for the variant of a delimited records format.
+      Variants specify a default set of parsing hints for how the delimited
+      file is formatted.  See the `records format specification
+      <https://github.com/bluelabsio/records-mover/blob/master/docs/RECORDS_SPEC.md>`_
+      for semantics of each.
+
+      alias of Literal[dumb, csv, bigquery, bluelabs, vertica]

docs/source/records_mover.records.sources.rst

@@ -0,0 +1,10 @@
+records\_mover.records.sources package
+======================================
+
+Module contents
+---------------
+
+.. automodule:: records_mover.records.sources
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/records_mover.records.targets.rst

@@ -0,0 +1,10 @@
+records\_mover.records.targets package
+======================================
+
+Module contents
+---------------
+
+.. automodule:: records_mover.records.targets
+   :members:
+   :undoc-members:
+   :show-inheritance:

docs/source/records_mover.rst

@@ -0,0 +1,26 @@
+records\_mover package
+======================
+
+Subpackages
+-----------
+
+.. toctree::
+   :maxdepth: 4
+
+   records_mover.airflow
+   records_mover.records
+
+Module contents
+---------------
+
+.. automodule:: records_mover
+   :members:
+   :undoc-members:
+   :show-inheritance:
+   :exclude-members: Session
+
+   .. autoclass:: records_mover.Session
+      :members: records, set_stream_logging, get_db_engine, get_default_db_engine
+      :undoc-members:
+      :show-inheritance:
+      :special-members: __init__

metrics/bigfiles_high_water_mark

@@ -1 +1 @@
-1002
+1019

metrics/coverage_high_water_mark

@@ -1 +1 @@
-93.700
+93.7100

metrics/flake8_high_water_mark

@@ -1 +1 @@
-174
+170

metrics/mypy_high_water_mark

@@ -1 +1 @@
-92.2800
+92.3100