Add docs helpers

LICENSE

@@ -0,0 +1,24 @@
+Copyright (c) 2017 Steemit, Inc., and contributors.
+
+The following license applies to code contained within this repository that
+is created by Steemit, Inc. Other copy right holders have licensed dependencies
+under their own individual licenses.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+3. The currency symbols, 'STEEM' and 'SBD' are not changed and no new currency symbols are added.
+4. The STEEMIT_INIT_PUBLIC_KEY_STR is not changed from STM8GC13uCZbP44HzMLV6zPZGwVQ8Nt4Kji8PapsPiNq1BK153XTX,
+and the software is not modified in any way that would bypass the need for the corresponding private to start
+a new blockchain.
+5. The software is not used with any forks of the Steem blockchain that are not recognized by Steemit, Inc in writing.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

Makefile

@@ -1,10 +1,13 @@
+ROOT_DIR := .
+DOCS_DIR := $(ROOT_DIR)/docs
+DOCS_BUILD_DIR := $(DOCS_DIR)/_build
 
 
 default: build
 
 .PHONY: test run
 
-build: pep8
+build: pep8 README.rst
 	docker build -t steemit/sbds .
 
 run:
@@ -17,8 +20,9 @@ pep8-test:
 	py.test --pep8 -m pep8
 
 fmt:
-	yapf --recursive .
-
-pep8:
-	pep8 .
-
+	yapf --recursive --in-place .
+	autopep8 --recursive --in-place .
+
+README.rst: docs/src/README.rst 
+	cd docs
+	$(MAKE) README

README.rst

@@ -1,61 +1,113 @@
-=================
-README for SBDS
-=================
 
-# Notice
+SBDS (Steem Blockchain Data Service)
+************************************
 
-This is prerelease software, not yet suitable for production use.  Proceed at your own risk.
 
-# Steem Blockchain Data Service
+Notice
+======
+
+This is prerelease software, not yet suitable for production use.
+Proceed at your own risk.
+
 
 Quickstart
 ==========
 
-`sbds` is available on Docker Hub as `steemit/sbds`.
+***sbds*** is available on Docker Hub as *steemit/sbds*.
 
 e.g.
 
-`docker run -d steemit/sbds`
+``docker run -d steemit/sbds``
+
 
-## Overview
+Overview
+========
 
 Stack: Python 3, SQLAlchemy, bottle.
 
-`sbds` is a set of tool for querying the data of the Steem Blockchain.
+**sbds** is a set of tool for querying the data of the Steem
+Blockchain.
+
+While providing direct interfaces to several pluggable storage
+architectures that may be used for querying the blockchain, **sbds**
+may also be used as a lower level API upon which other applications
+can be built.
 
-While providing direct interfaces to several pluggable storage architectures that may be used for querying the blockchain, `sbds` may also be used as a lower level API upon which other applications can be built.
 
 Architecture
 ============
 
 The system has three conceptual functions:
 
-1. Interfacing with a `steemd` instance to provide access to blocks, ranges of blocks, or the continual stream of
-blocks as they are published on the blockchain.
+1. Interface with a *steemd* instance to provide access to blocks,
+ranges of blocks, or the continual stream of blocks as they are
+published on the blockchain.
+
+1. Ingest, prepare, store, and index blocks in one of two storage
+   backends (S3, SQL Database).
 
-2. Ingest, prepare, store, and index blocks in one of 3 storage backends (S3, SQL Database, and/or Elasticsearch).
+2. Querying indexed blocks.
 
-3. Querying indexed blocks.
 
 Install
 =======
 
-`sbds` is an installable python 3 package, though it is currently not published on pipy, and must be installed using git:
+**sbds** is an installable python 3 package, though it is currently
+not published on pipy, and must be installed using git:
+
+``pip3 install -e [email protected]:steemit/sbds.git#egg=sbds``
+
+Installation will (during early development) require mysql and
+postgres development sources in order to build correctly. As an
+alternative to installing those libraries, a *Dockerfile* is
+available.
+
+
+Usage
+=====
+
+On initial use, blocks can be quickly loaded from "checkpoints" which
+are gzipped text files that are 1M blocks in length and currently
+hosted on S3 at   ``s3://steemit-dev-sbds-checkpoints``.
+
+Once the storage is synced with all previous blocks, blocks can be
+streamed to storage backends as they are confirmed.
+
+These blocks are not cryptographically assured in any way (and
+**sbds** does not provide any cryptographic guarantees or verify
+blockchain consensus state), so you may wish to regenerate these
+checkpoints.
+
+**sbds** is designed to always be used in conjunction with a trusted
+instance of *steemd* to validate all block data before **sbds** ever
+receives it.  This daemon **does not** implement any consensus rules.
+
+
+CLI
+====
+
+The **sbds** package installs the ``sbds`` CLI.
+
+More information:
 
-`pip3 install -e [email protected]:steemit/sbds.git#egg=sbds`
+::
 
-Installation will (during early development) require mysql and postgres development sources in order to build
-correctly. As an alternative to installing those libraries, a `Dockerfile` is available.
+   $ sbds --help
+   Usage: sbds [OPTIONS] COMMAND [ARGS]...
 
-## Usage
+     The *sbds* CLI manages storage, retrieval, and querying of the Steem
+     blockchain.
 
-On initial use, blocks can be quickly loaded from "checkpoints" which are gzipped text files that are 1M
-blocks in length and currently hosted on S3 at `s3://steemit-dev-sbds-checkpoints`.
+     sbds has several commands, each of which has additional subcommands.
 
-Once the storage is synced with all previous blocks, blocks can be streamed to storage backends as they are confirmed.
+     For more detailed information on a command and its flags, run:
+         sbds COMMAND --help
 
-These blocks are not cryptographically assured in any way (and `sbds` does not provide any cryptographic guarantees
-or verify blockchain consensus state), so you may wish to regenerate these checkpoints.
+   Options:
+     --help  Show this message and exit.
 
-`sbds` is designed to always be used in conjunction with a trusted instance of `steemd` to validate all block data before
-`sbds` ever receives it.  This daemon **does not** implement any consensus rules.
+   Commands:
+     chain        Query the Steem blockchain
+     checkpoints  retrieve blocks from blockchain checkpoints
+     db           Interact with an SQL storage backend
+     s3           Interact with an S3 storage backend

docs/Makefile

@@ -7,12 +7,19 @@ SPHINXBUILD   = sphinx-build
 SPHINXPROJ    = sbds
 SOURCEDIR     = .
 BUILDDIR      = _build
+_README        = ../README.rst
 
 # Put it first so that "make" without argument is like "make help".
 help:
 	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
 
-.PHONY: help Makefile
+.PHONY: help Makefile README
+
+README:
+	$(MAKE) clean
+	$(MAKE) rst
+	cp "$(BUILDDIR)"/rst/src/README.rst ..
+
 
 # Catch-all target: route all unknown targets to Sphinx using the new
 # "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).

docs/conf.py

@@ -16,12 +16,27 @@
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the
 # documentation root, use os.path.abspath to make it absolute, like shown here.
-#
+
+
 import os
 import sys
 sys.path.insert(0, os.path.abspath('..'))
 sys.path.insert(0, os.path.abspath('_themes'))
+from recommonmark.parser import CommonMarkParser
+from recommonmark.transform import AutoStructify
+
 import sbds
+# setup AutoStructify
+def setup(app):
+    app.add_config_value('recommonmark_config', {
+            'url_resolver': lambda url: github_doc_root + url,
+            'auto_toc_tree_section': 'Contents',
+            }, True)
+    app.add_transform(AutoStructify)
+
+
+
+
 
 # -- General configuration ------------------------------------------------
 
@@ -32,18 +47,27 @@
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = ['sphinx.ext.autodoc',
+extensions = [
+
+    'sphinx.ext.autodoc',
     'sphinx.ext.coverage',
     'sphinx.ext.ifconfig',
     'sphinx.ext.viewcode',
-    'sphinx.ext.githubpages']
+    'sphinx.ext.githubpages',
+    'sphinxcontrib.programoutput',
+    'sphinxcontrib.restbuilder',
+    ]
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
 # The suffix(es) of source filenames.
 # You can specify multiple suffix as a list of string:
 #
+# https://github.com/rtfd/recommonmark
+source_parsers = {
+    '.md': CommonMarkParser,
+}
 source_suffix = ['.rst', '.md']
 #source_suffix = '.rst'
 

docs/index.rst

@@ -9,8 +9,7 @@ Welcome to sbds's documentation!
 .. toctree::
    :maxdepth: 2
    :caption: Contents:
-   :glob:
-    src/*
+   :glob: src/*
 
 
 Indices and tables

docs/src/README.rst

@@ -0,0 +1,74 @@
+===============================================
+SBDS (Steem Blockchain Data Service)
+===============================================
+
+Notice
+======
+
+This is prerelease software, not yet suitable for production use.  Proceed at your own risk.
+
+
+Quickstart
+==========
+
+***sbds*** is available on Docker Hub as `steemit/sbds`.
+
+e.g.
+
+``docker run -d steemit/sbds``
+
+Overview
+========
+
+Stack: Python 3, SQLAlchemy, bottle.
+
+**sbds** is a set of tool for querying the data of the Steem Blockchain.
+
+While providing direct interfaces to several pluggable storage architectures that may be used for querying the blockchain, **sbds** may also be used as a lower level API upon which other applications can be built.
+
+Architecture
+============
+
+The system has three conceptual functions:
+
+1. Interfacing with a `steemd` instance to provide access to blocks, ranges of blocks, or the continual stream of
+blocks as they are published on the blockchain.
+
+2. Ingest, prepare, store, and index blocks in one of two storage backends (S3, SQL Database, and/or Elasticsearch).
+
+3. Querying indexed blocks.
+
+Install
+=======
+
+**sbds** is an installable python 3 package, though it is currently not published on pipy, and must be installed using git:
+
+``pip3 install -e [email protected]:steemit/sbds.git#egg=sbds``
+
+Installation will (during early development) require mysql and postgres development sources in order to build
+correctly. As an alternative to installing those libraries, a `Dockerfile` is available.
+
+Usage
+=====
+
+On initial use, blocks can be quickly loaded from "checkpoints" which are gzipped text files that are 1M
+blocks in length and currently hosted on S3 at   ``s3://steemit-dev-sbds-checkpoints``.
+
+Once the storage is synced with all previous blocks, blocks can be streamed to storage backends as they are confirmed.
+
+These blocks are not cryptographically assured in any way (and **sbds** does not provide any cryptographic guarantees
+or verify blockchain consensus state), so you may wish to regenerate these checkpoints.
+
+**sbds** is designed to always be used in conjunction with a trusted instance of `steemd` to validate all block data before
+**sbds** ever receives it.  This daemon **does not** implement any consensus rules.
+
+CLI
+===
+The **sbds** package installs the ``sbds`` CLI.
+
+More information
+
+.. command-output:: sbds --help
+
+
+

docs/src/cli.rst

@@ -0,0 +1,7 @@
+.. command-output:: sbds chain --help
+
+.. command-output:: sbds chain block_height --help
+
+.. command-output:: sbds chain get-blocks --help
+
+.. command-output:: sbds chain stream-blocks --help