Create, publish public API documentation on readthedocs.io #98

vinceatbluelabs · 2020-07-08T17:48:37Z

This documents a minimal public API for records-mover at https://records-mover.readthedocs.io/en/publish_docs/

setup.py

351: tests/integration/itest 349: setup.py 320: records_mover/records/schema/field/__init__.py Reduce total number of bigfiles violations to 1002 or below!

vinceatbluelabs · 2020-07-09T13:42:54Z

README.md

@@ -2,6 +2,8 @@
 src="https://raw.githubusercontent.com/bluelabsio/records-mover/master/docs/records-mover-horizontal.png"
 alt="Records Mover">

+[![Documentation Status](https://readthedocs.org/projects/records-mover/badge/?version=publish_docs)](https://records-mover.readthedocs.io/en/publish_docs/?badge=latest)


Update readthedocs URLs back to master branch

vinceatbluelabs · 2020-07-09T13:43:43Z

deps.sh

@@ -20,4 +20,4 @@ pip3 install --upgrade pip
 # It's nice to unit test, integration test, and run the CLI in
 # a development pyenv.
 #
-pip3 install -r requirements.txt -e '.[unittest,itest,cli,typecheck]'
+pip3 install -r requirements.txt -e '.[unittest,itest,cli,typecheck,docs]'


With this extra, you can run 'make docs' in the docs/ directory and generate things locally.

vinceatbluelabs · 2020-07-09T13:48:39Z

docs/source/index.rst

+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`


I was sort of expecting documentation generation to be entirely driven by docstring comments, but the Sphinx toolkit seems to be a tad more rustic than that. As a result, these .rst files contain:

any tweaks that need to be done to the default generation strategy (you'll see some examples later on)

lists of the specific public API that will be documented

any module-level documentation needed, which doesn't seem to be brought in at all (I generally chose to leave this out and preferred documenting things at the class level)

vinceatbluelabs · 2020-07-09T13:55:08Z

docs/source/records_mover.records.rst

+   :maxdepth: 4
+
+   records_mover.records.sources
+   records_mover.records.targets


Most of the 'documenting only the public API' comes from judicious exclusion of things from these subpackage lists.

vinceatbluelabs · 2020-07-09T13:57:24Z

docs/source/records_mover.records.rst

+      <https://github.com/bluelabsio/records-mover/blob/master/docs/RECORDS_SPEC.md>`_
+      for semantics of each.
+
+      alias of Literal[dumb, csv, bigquery, bluelabs, vertica]


Sadly, there's no way to assign a docstring to:

DelimitedVariant = Literal[dumb, csv, bigquery, bluelabs, vertica]

...so I had to do it out-of-band here.

vinceatbluelabs · 2020-07-09T14:01:07Z

docs/source/records_mover.records.rst

+.. autoclass:: records_mover.records.base_records_format.BaseRecordsFormat
+   :members:
+   :show-inheritance:
+   :special-members: __init__


Most of the time I chose to bring up key public API classes into the __all__ of records_mover or record_mover.records, which also ensures that Sphinx will document it there. I then left out the implementation module where it was defined from Sphinx so it didn't get double-documented, and so that we can move things around without folks doing deep 'import' statements.

This is a case where I couldn't get away with that, as Sphinx documented it as a base class of DelimitedRecordsFormat and ParquetRecordsFormat--leaving this out broke the link. I couldn't figure how to convince it to do otherwise.

vinceatbluelabs · 2020-07-09T14:05:05Z

docs/source/records_mover.rst

+   :exclude-members: Session
+
+   .. autoclass:: records_mover.Session
+      :members: records, set_stream_logging, get_db_engine, get_default_db_engine


I documented only specific methods here. In particular I didn't yet document the Creds hierarchy - it seemed like a whole different can of worms to lock down at this point. As a result, the methods needed to do google sheets work aren't documented in Sphinx yet - I dropped a post-1.0 backlog item to deal with that.

vinceatbluelabs · 2020-07-09T14:05:31Z

metrics/bigfiles_high_water_mark

@@ -1 +1 @@
-1002
+1019


setup.py has more extras for documentation generation.

vinceatbluelabs · 2020-07-09T15:20:36Z

records_mover/records/__init__.py

    'Records',
-    'move'
+    'move',
+    'MoveResult'


Let's bring more things so they can be pulled in with a simple import from records_mover.records.

vinceatbluelabs · 2020-07-09T15:21:12Z

records_mover/records/mover.py

+    # the first method which makes sense for your new source/target.
+    # See documentation for RecordsSource in sources.py and
+    # RecordsTarget in targets.py for the semantics of the methods
+    # being called.


Let's leave the maintenance documentation out of the public API documentation.

vinceatbluelabs · 2020-07-09T15:22:10Z

records_mover/records/schema/__init__.py

+    'RecordsSchemaFieldConstraints',
+    'RecordsSchemaFieldStatistics',
+    'RecordsSchemaFieldRepresentation',
+]


I started down the path of bringing this stuff into the public API but ultimately punted. At least we can have simpler imports of it in the meantime.

vinceatbluelabs · 2020-07-09T15:22:59Z

records_mover/records/types.py

 RecordsFormatType = Literal['delimited', 'parquet']

-VALID_VARIANTS = ['dumb', 'csv', 'bigquery', 'bluelabs', 'vertica']


This thing was only used in unit tests, and can now be done with typing_inspect anyway.

vinceatbluelabs · 2020-07-09T15:23:16Z

records_mover/records/types.py

-#
-# Note: Any expansion of these types should also be done in
-# records.jobs.hints
-#


No longer true! These are all sourced from one place now.

vinceatbluelabs added 30 commits June 29, 2020 13:04

Add sphinx dependencies

6f4b13f

Add initial docs source

61a790f

Add records_mover to toctree

3abfab2

Autogenerate source apidoc rst files

be2a095

Allow running after already cleared

a74c47b

Deal with issue during apidocs import

c444f3c

Revert automatic generation - readthedocs.io does not support

52197e7

readthedocs/readthedocs.org#1139

Drop submodules, include special-members

98812c5

Document public API of records_mover.airflow

58d3f95

Move GoogleCloudCredentialsHook to airflow.hooks

b18873a

Document public API of records_mover.airflow

66ce59c

Trim things which are not part of public API

0d1f78d

Exclude RecordsSchema from public documentation for now

24bc882

Fix formatting

63b437f

Fix some documentation formatting

0f63d74

Merge remote-tracking branch 'origin/master' into publish_docs

dfd93ae

Format sources factory

9b1995a

Tweak records target documentation

7131ad1

Tweak records target documentation

5872e52

Clear out records submodule documentation

bf783d5

Use get_args() instead of VALID_VARIANTS

a563aa3

Document more things

fd978cb

Create docs extra

cb018a6

Create docs extra

d25f27a

Fix syntax

1979dd9

Create docs extra

40d5958

Create docs extra

280a981

Create docs extra

23e89f9

Fix missing TOCs, remove RecordsSchema

7fe9510

Have Sphinx fail on warning

20411b8

bluelabsbutler reviewed Jul 8, 2020

View reviewed changes

setup.py Outdated Show resolved Hide resolved

vinceatbluelabs added 9 commits July 8, 2020 16:37

Bump bigfiles

53a9897

351: tests/integration/itest 349: setup.py 320: records_mover/records/schema/field/__init__.py Reduce total number of bigfiles violations to 1002 or below!

Fix flake8 issues

6d1fc2f

Ratchet

80dbd1d

Fix mypy errors

7eb2960

Ratchet coverage

e16b177

Fix tests around DelimitedVariant docs

74a4838

Unrachet

8bd46bb

Ratchet

788a9b3

Add trailing newline

9ec6984

vinceatbluelabs commented Jul 9, 2020

View reviewed changes

Fix doc drift

93f672e

vinceatbluelabs commented Jul 9, 2020

View reviewed changes

vinceatbluelabs added 3 commits July 9, 2020 09:50

Clarify comment

e513abc

Clarify why docstring is missing

2ba42e4

Clarify doc generation

37c2b62

vinceatbluelabs commented Jul 9, 2020

View reviewed changes

metrics/bigfiles_high_water_mark

@@ -1 +1 @@

1002

1019

Copy link

Contributor Author

vinceatbluelabs Jul 9, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

setup.py has more extras for documentation generation.

vinceatbluelabs commented Jul 9, 2020

View reviewed changes

vinceatbluelabs requested a review from cwegrzyn July 9, 2020 15:23

vinceatbluelabs merged commit 3312802 into master Jul 13, 2020

vinceatbluelabs deleted the publish_docs branch July 13, 2020 13:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create, publish public API documentation on readthedocs.io #98

Create, publish public API documentation on readthedocs.io #98

vinceatbluelabs commented Jul 8, 2020 •

edited

Loading

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

vinceatbluelabs Jul 9, 2020

		RecordsFormatType = Literal['delimited', 'parquet']

		VALID_VARIANTS = ['dumb', 'csv', 'bigquery', 'bluelabs', 'vertica']

Create, publish public API documentation on readthedocs.io #98

Create, publish public API documentation on readthedocs.io #98

Conversation

vinceatbluelabs commented Jul 8, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vinceatbluelabs commented Jul 8, 2020 •

edited

Loading