Add Sphinx documentation framework (project-chip#24185)

* doc: add sphinx documentation framework

Added the basics for a sphinx-based doc portal.

Added an extension `external_content` that copies the relevant
doc files that are used in the documentation. This extension
also modifies files while copying to adjust links and allow
syntax that renders on both GitHub and Sphinx.

Warnings when building the documentation have been
fixed or removed. Fixes include:
- Linking errors
- Typos
- ToC tree errors
- Adding MyST configuration data
- Removing anchors on the form <a name="my-anchor"></a> as they
  cannot be used by sphinx
- Changed the GitHub rendered emoji `:heavy_check_mark:` to the
ascii symbol ✔ so that it renders with Sphinx

Signed-off-by: Gaute Svanes Lunde <[email protected]>

* workflows: expand markdown fences in spellchecker

Expand the markdown fences to include MyST for Sphinx syntax
like the following:

```{include} my/file.md
```

Signed-off-by: Gaute Svanes Lunde <[email protected]>

* workflows: add docbuild workflow

Add a workflow for building the documentation with Sphinx and
deploy the generated html to the
`project-chip/connectedhomeip-doc` repository to be used by
github.io. This will overwrite the existing Doxygen
documentation that is hosted there.

Signed-off-by: Gaute Svanes Lunde <[email protected]>

* Restyle

* doc: move link targets to build folder

Created a change in the `external_content` extension so that
links to content that is also being copied to the _build folder
will point to their new location instead. This enables linking
to sections on other pages.

This change also enables link checking for sections on other
pages, and therefore also includes a fix to some previously
broken section links.

Signed-off-by: Gaute Svanes Lunde <[email protected]>

Signed-off-by: Gaute Svanes Lunde <[email protected]>
Co-authored-by: Andrei Litvin <[email protected]>

.github/workflows/docbuild.yaml

@@ -0,0 +1,49 @@
+name: Documentation Build
+
+on:
+  push:
+    branches:
+      - master
+
+permissions:
+  contents: write
+
+jobs:
+  build-and-deploy:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout the code
+        uses: actions/checkout@v2
+        with:
+          path: matter
+          fetch-depth: 0
+      - name: Install Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.8
+      - name: cache-pip
+        uses: actions/cache@v1
+        with:
+          path: ~/.cache/pip
+          key: ${{ runner.os }}-doc-pip
+      - name: Install base dependencies
+        working-directory: matter
+        run: |
+          sudo pip3 install -U pip
+          pip3 install -r docs/requirements.txt
+      - name: Build documentation
+        working-directory: matter/docs
+        run: |
+          mkdir -p _build/src
+          make html
+          touch _build/html/.nojekyll
+      - name: Deploy to gh-pages
+        if: github.repository == 'project-chip/connectedhomeip'
+        uses: peaceiris/actions-gh-pages@v3
+        with:
+          deploy_key: ${{ secrets.DOXYGEN_DEPLOY_KEY }}
+          external_repository: project-chip/connectedhomeip-doc
+          publish_dir: matter/docs/_build/html
+          # Keep only the latest version of the documentation
+          force_orphan: true

.gitignore

@@ -50,6 +50,12 @@ __pycache__
 # Doxygen outputs
 docs/html
 
+# Python venv
+.venv
+
+# Documentation
+docs/_build
+
 # VSCode java extensions
 .project
 

.spellcheck.yml

@@ -46,7 +46,11 @@ matrix:
           # ```python
           # content
           # ```
-          - open: '(?s)^(?P<open> *`{3,})[a-z]*$'
+          #
+          # Allow MyST extended syntax like:
+          # ```{include} my/file.md
+          # ```
+          - open: '(?s)^(?P<open> *`{3,})([a-z]*$|{[a-z]*?}\s*[^\n]*?$)'
             close: '^(?P=open)$'
           # Ignore text between inline back ticks
           - open: '(?P<open>`+)'

docs/Doxyfile

@@ -830,7 +830,7 @@ INPUT                  = README.md \
                          docs/guides/BUILDING.md \
                          docs/VSCODE_DEVELOPMENT.md \
                          docs/PROJECT_FLOW.md \
-                         docs/STYLE_GUIDE.md \
+                         docs/style/style_guide.md \
                          src/ble \
                          src/controller \
                          src/crypto \

docs/ERROR_CODES.md

@@ -8,7 +8,7 @@ This file was **AUTOMATICALLY** generated by
 -   [SDK Core errors: range `0x000..0x0FF`](#sdk-core-errors)
 -   [SDK Inet Layer errors: range `0x100..0x1FF`](#sdk-inet-layer-errors)
 -   [SDK Device Layer errors: range `0x200..0x2FF`](#sdk-device-layer-errors)
--   [ASN.1 Layer errors: range `0x300..0x3FF`](#asn-1-layer-errors)
+-   [ASN.1 Layer errors: range `0x300..0x3FF`](#asn.1-layer-errors)
 -   [BLE Layer errors: range `0x400..0x4FF`](#ble-layer-errors)
 -   [IM Global errors errors: range `0x500..0x5FF`](#im-global-errors-errors)
 

docs/Makefile

@@ -0,0 +1,21 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?= -W -c . -d _build/doctrees
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = _build/src
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	mkdir -p "$(SOURCEDIR)"
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/README.md

@@ -24,7 +24,7 @@
 ## Style Guide
 
 -   Documentation about style is documented in
-    [the style guide](./STYLE_GUIDE.md)
+    [the style guide](./style/style_guide.md)
 -   Additional documentation about more specific files are in the
     [style folder](./style/)
 

docs/_extensions/external_content.py

@@ -0,0 +1,251 @@
+"""
+External content
+################
+
+Copyright (c) 2021 Nordic Semiconductor ASA
+SPDX-License-Identifier: Apache-2.0
+
+Introduction
+============
+
+This extension allows to import sources from directories out of the Sphinx
+source directory. They are copied to the source directory before starting the
+build. Note that the copy is *smart*, that is, only updated files are actually
+copied. Therefore, incremental builds detect changes correctly and behave as
+expected.
+
+Links to external content not included in the generated documentation are
+transformed to external links as needed.
+
+Configuration options
+=====================
+
+- ``external_content_contents``: A list of external contents. Each entry is
+  a tuple with two fields: the external base directory and a file glob pattern.
+- ``external_content_link_prefixes``: A list of link prefixes out of scope.
+  All links to content with these prefixes are made external.
+- ``external_content_link_extensions``: A list of file extensions in scope of
+  the documentation. All links to content without these file extensions are
+  made external.
+- ``external_content_keep``: A list of file globs (relative to the destination
+  directory) that should be kept even if they do not exist in the source
+  directory. This option can be useful for auto-generated files in the
+  destination directory.
+"""
+
+import filecmp
+import os
+from pathlib import Path
+import re
+import shutil
+import tempfile
+from typing import Dict, Any, List, Optional
+
+from sphinx.application import Sphinx
+
+__version__ = "0.1.0"
+
+DIRECTIVES = ("figure", "image", "include", "literalinclude")
+"""Default directives for included content."""
+
+EXTERNAL_LINK_URL_PREFIX = (
+    "https://github.com/project-chip/connectedhomeip/blob/master/"
+)
+
+
+def adjust_includes(
+    fname: Path,
+    basepath: Path,
+    encoding: str,
+    link_prefixes: List[str],
+    extensions: List[str],
+    targets: List[Path],
+    dstpath: Optional[Path] = None,
+) -> None:
+    """Adjust included content paths.
+
+    Args:
+        fname: File to be processed.
+        basepath: Base path to be used to resolve content location.
+        encoding: Sources encoding.
+        link_prefixes: Prefixes of links that are made external.
+        extensions: Filename extensions links to which are not made external.
+        targets: List of all files that are being copied.
+        dstpath: Destination path for fname if its path is not the actual destination.
+    """
+
+    if fname.suffix != ".md":
+        return
+
+    dstpath = dstpath or fname.parent
+
+    def _adjust_path(path):
+        # ignore absolute paths, section links, hyperlinks and same folder
+        if path.startswith(("/", "#", "http", "www")) or not "/" in path:
+            return path
+
+        # for files that are being copied modify reference to and out of /docs
+        filepath = path.split("#")[0]
+        absolute = (basepath / filepath).resolve()
+        if absolute in targets:
+            if "docs/" in path:
+                path = path.replace("docs/", "")
+            elif "../examples" in path:
+                path = path.replace("../", "", 1)
+            return path
+
+        # otherwise change links to point to their targets' original location
+        return Path(os.path.relpath(basepath / path, dstpath)).as_posix()
+
+    def _adjust_links(m):
+        displayed, fpath = m.groups()
+        fpath_adj = _adjust_path(fpath)
+        return f"[{displayed}]({fpath_adj})"
+
+    def _adjust_external(m):
+        displayed, target = m.groups()
+        return f"[{displayed}]({EXTERNAL_LINK_URL_PREFIX}{target})"
+
+    def _adjust_filetype(m):
+        displayed, target, extension = m.groups()
+        if extension.lower() in extensions or target.startswith("http"):
+            return m.group(0)
+
+        return f"[{displayed}]({EXTERNAL_LINK_URL_PREFIX}{target})"
+
+    def _adjust_image_link(m):
+        prefix, fpath, postfix = m.groups()
+        fpath_adj = _adjust_path(fpath)
+        return f"{prefix}{fpath_adj}{postfix}"
+
+    rules = [
+        # Find any links and adjust the path
+        (r"\[([^\[\]]*)\]\s*\((.*)\)", _adjust_links),
+
+        # Find links that lead to an external folder and transform it
+        # into an external link.
+        (
+            r"\[([^\[\]]*)\]\s*\((?:\.\./)*((?:" + "|".join(link_prefixes) + r")[^)]*)\)",
+            _adjust_external,
+        ),
+
+        # Find links that lead to a non-presentable filetype and transform
+        # it into an external link.
+        (
+            r"\[([^\[\]]*)\]\s*\((?:\.\./)*((?:[^()]+?/)*[^.()]+?(\.[^)/#]+))(?:#[^)]+)?\)",
+            _adjust_filetype,
+        ),
+
+        # Find links that lead to a folder and transform it into an external link.
+        (
+            r"\[([^\[\]]*)\]\s*\((?:\.\./)*((?:[^()]+?/)+[^).#/]+)(\))",
+            _adjust_filetype,
+        ),
+
+        # Find image links in img tags and adjust them
+        (r"(<img [^>]*src=[\"'])([^ >]+)([\"'][^>]*>)", _adjust_image_link)
+    ]
+
+    with open(fname, "r+", encoding=encoding) as f:
+        content = f.read()
+        modified = False
+
+        for pattern, sub_func in rules:
+            content, changes_made = re.subn(pattern, sub_func, content)
+            modified = modified or changes_made
+
+        if modified:
+            f.seek(0)
+            f.write(content)
+            f.truncate()
+
+
+def sync_contents(app: Sphinx) -> None:
+    """Synhronize external contents.
+
+    Args:
+        app: Sphinx application instance.
+    """
+
+    srcdir = Path(app.srcdir).resolve()
+    to_copy = []
+    to_delete = set(f for f in srcdir.glob("**/*") if not f.is_dir())
+    to_keep = set(
+        f
+        for k in app.config.external_content_keep
+        for f in srcdir.glob(k)
+        if not f.is_dir()
+    )
+
+    for content in app.config.external_content_contents:
+        prefix_src, glob = content
+        for src in prefix_src.glob(glob):
+            if src.is_dir():
+                to_copy.extend(
+                    [(f, prefix_src) for f in src.glob("**/*") if not f.is_dir()]
+                )
+            else:
+                to_copy.append((src, prefix_src))
+
+    list_of_destinations = [f for f, _ in to_copy]
+
+    for entry in to_copy:
+        src, prefix_src = entry
+        dst = (srcdir / src.relative_to(prefix_src)).resolve()
+
+        if dst in to_delete:
+            to_delete.remove(dst)
+
+        if not dst.parent.exists():
+            dst.parent.mkdir(parents=True)
+
+        # just copy if it does not exist
+        if not dst.exists():
+            shutil.copy(src, dst)
+            adjust_includes(
+                dst,
+                src.parent,
+                app.config.source_encoding,
+                app.config.external_content_link_prefixes,
+                app.config.external_content_link_extensions,
+                list_of_destinations,
+            )
+        # if origin file is modified only copy if different
+        elif src.stat().st_mtime > dst.stat().st_mtime:
+            with tempfile.TemporaryDirectory() as td:
+                # adjust origin includes before comparing
+                src_adjusted = Path(td) / src.name
+                shutil.copy(src, src_adjusted)
+                adjust_includes(
+                    src_adjusted,
+                    src.parent,
+                    app.config.source_encoding,
+                    app.config.external_content_link_prefixes,
+                    app.config.external_content_link_extensions,
+                    list_of_destinations,
+                    dstpath=dst.parent,
+                )
+
+                if not filecmp.cmp(src_adjusted, dst):
+                    dst.unlink()
+                    shutil.move(os.fspath(src_adjusted), os.fspath(dst))
+
+    # remove any previously copied file not present in the origin folder,
+    # excepting those marked to be kept.
+    for file in to_delete - to_keep:
+        file.unlink()
+
+
+def setup(app: Sphinx) -> Dict[str, Any]:
+    app.add_config_value("external_content_contents", [], "env")
+    app.add_config_value("external_content_keep", [], "")
+    app.add_config_value("external_content_link_prefixes", [], "env")
+    app.add_config_value("external_content_link_extensions", [], "env")
+
+    app.connect("builder-inited", sync_contents)
+
+    return {
+        "version": __version__,
+        "parallel_read_safe": True,
+        "parallel_write_safe": True,
+    }

docs/api/index.md

@@ -0,0 +1,7 @@
+# API
+
+```{toctree}
+:glob:
+
+*
+```