Skip to content

Commit

Permalink
Initial Sphinx revamp (#2459)
Browse files Browse the repository at this point in the history
* Remove unused docs/package.json

Nowadays sphinx-autobuild offers similar functionality.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Fix Sphinx language config

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Fix circular imports

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Remove unused documentation extensions

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Try to not move files around

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Fix CSS paths

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Fix file hierarchy

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Fix Read the Docs configuration

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Update documentation dependencies

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Manually fix JupyterCommandGroup API docs

See gh-2453.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Install all documentation dependencies

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Restore test requirements

kedro.extras.datasets is still being tested.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Remove all autosummary members from problematic class

See gh-2453.

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Temporarily disable nitpicky mode for linkcheck

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Upgrade Read the Docs to Python 3.8

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Fix destination of linkcheck

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Minor fixes to docstrings

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Minor reference fix

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Cap sphinx-autodoc-typehints version

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Restore nitpicky linkcheck

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Restore treating documentation warnings as errors

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Remove redundant build-docs CI job

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Ignore .rst files generated to support docs build

Signed-off-by: Jo Stichbury <[email protected]>

* Explain ignored documentation files

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Move linkcheck to Read the Docs

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Remove old devserver Makefile target

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Fix databricks broken reference

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Fix pyspark broken reference

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

* Fix other broken references

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>

---------

Signed-off-by: Juan Luis Cano Rodríguez <[email protected]>
Signed-off-by: Jo Stichbury <[email protected]>
Co-authored-by: Jo Stichbury <[email protected]>
  • Loading branch information
astrojuanlu and stichbury authored Mar 30, 2023
1 parent c2968ba commit 38dfe6f
Show file tree
Hide file tree
Showing 48 changed files with 61 additions and 190 deletions.
53 changes: 0 additions & 53 deletions .circleci/continue_config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -354,34 +354,6 @@ jobs:
name: Pip-compile requirements file
command: conda activate kedro_builder; make pip-compile

build_docs:
executor:
name: docker
python_version: "3.7"
steps:
- setup
- run:
name: Build docs
command: make build-docs
- run:
name: Pip freeze including docs dependencies
command: pip freeze
when: always

docs_linkcheck:
executor:
name: docker
python_version: "3.8"
steps:
- setup
- run:
name: Check for broken links
command: make linkcheck
- run:
name: Pip freeze including docs dependencies
command: pip freeze
when: always

sync:
docker:
# https://circleci.com/docs/2.0/circleci-images/#circleci-base-image
Expand Down Expand Up @@ -544,27 +516,6 @@ jobs:
workflows:
version: 2.1

build_docs_only:
when:
and:
- <<pipeline.parameters.docs_change>>
- not: <<pipeline.parameters.code_change>>
- not: <<pipeline.parameters.release_kedro>>
- not: <<pipeline.parameters.run_hourly>>
- not: <<pipeline.parameters.run_nightly>>
jobs:
- build_docs
- docs_linkcheck
- lint:
matrix:
parameters:
python_version: ["3.7", "3.8", "3.9", "3.10"]
- all_circleci_checks_succeeded:
requires:
- build_docs
- docs_linkcheck
- lint

build_code_and_docs:
when:
and:
Expand Down Expand Up @@ -601,8 +552,6 @@ workflows:
matrix:
parameters:
python_version: ["3.7", "3.8", "3.9", "3.10"]
- build_docs
- docs_linkcheck
- all_circleci_checks_succeeded:
requires:
- e2e_tests
Expand All @@ -612,8 +561,6 @@ workflows:
- lint
- pip_compile
- win_pip_compile
- build_docs
- docs_linkcheck

main_updated:
when:
Expand Down
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -134,6 +134,11 @@ venv.bak/
/site
/kedro/framework/html

# Sphinx documentation
# Additional files created by sphinx.ext.autosummary
# Some of them are actually tracked to control the output
/docs/source/kedro.*

# mypy
.mypy_cache/

Expand Down
8 changes: 5 additions & 3 deletions .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,20 +9,22 @@ version: 2
build:
os: ubuntu-22.04
tools:
python: "3.7"
python: "3.8"
nodejs: "19"
apt_packages:
- libasound2
jobs:
post_create_environment:
- npm install -g @mermaid-js/mermaid-cli
- ./docs/kedro-datasets-docs.sh
pre_build:
- python -m sphinx -WETan -j auto -D language=en -b linkcheck -d _build/doctrees docs/source _build/linkcheck

# Build documentation in the docs/ directory with Sphinx
sphinx:
builder: html
configuration: docs/conf.py
fail_on_warning: false # Turn back on soon
configuration: docs/source/conf.py
fail_on_warning: true

# Build documentation with MkDocs
# mkdocs:
Expand Down
3 changes: 0 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,6 @@ linkcheck:
pip install -e ".[docs]"
./docs/build-docs.sh "linkcheck"

devserver: build-docs
cd docs && npm install && npm start

package: clean install
python setup.py sdist bdist_wheel

Expand Down
29 changes: 3 additions & 26 deletions docs/build-docs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,35 +7,12 @@ set -o nounset

action=$1

# Move some files around. We need a separate build directory, which would
# have all the files, build scripts would shuffle the files,
# we don't want that happening on the actual code locally.
# When running on ReadTheDocs, sphinx-build would run directly on the original files,
# but we don't care about the code state there.
rm -rf docs/build
# Reinstall kedro-datasets locally
rm -rf kedro/datasets
mkdir docs/build/
cp -r docs/_templates docs/conf.py docs/*.svg docs/*.json docs/build/

bash docs/kedro-datasets-docs.sh

if [ "$action" == "linkcheck" ]; then
sphinx-build -c docs/ -ETan -j auto -D language=en -b linkcheck docs/build/ docs/build/html
sphinx-build -WETan -j auto -D language=en -b linkcheck -d docs/build/doctrees docs/source docs/build/linkcheck
elif [ "$action" == "docs" ]; then
sphinx-build -c docs/ -ETa -j auto -D language=en docs/build/ docs/build/html
sphinx-build -WETa -j auto -D language=en -b html -d docs/build/doctrees docs/source docs/build/html
fi

# Clean up build artefacts
rm -rf docs/build/html/_sources

# Copy built HTML to temp directory, clean up build dir and replace with built docs only
rm -rf docs/temp
mkdir docs/temp/
mkdir docs/temp/html
cp -rf docs/build/html/* docs/temp/html

rm -rf docs/build
mkdir docs/build
mkdir docs/build/html
cp -rf docs/temp/html/* docs/build/html
rm -rf docs/temp
16 changes: 0 additions & 16 deletions docs/package.json

This file was deleted.

File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
43 changes: 3 additions & 40 deletions docs/conf.py → docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,9 +15,7 @@
import importlib
import os
import re
import shutil
import sys
from distutils.dir_util import copy_tree
from inspect import getmembers, isclass, isfunction
from pathlib import Path
from typing import List, Tuple
Expand Down Expand Up @@ -49,12 +47,8 @@
"sphinx.ext.napoleon",
"sphinx_autodoc_typehints",
"sphinx.ext.doctest",
"sphinx.ext.todo",
"sphinx.ext.coverage",
"sphinx.ext.mathjax",
"sphinx.ext.ifconfig",
"sphinx.ext.viewcode",
"nbsphinx",
"sphinx_copybutton",
"sphinxcontrib.mermaid",
"myst_parser",
Expand All @@ -68,6 +62,7 @@

# Add any paths that contain templates here, relative to this directory.
templates_path = ["_templates"]
html_static_path = ["_static"]

# The suffix(es) of source filenames.
# You can specify multiple suffix as a list of string:
Expand All @@ -82,7 +77,7 @@
#
# This is also used if you do content translation via gettext catalogs.
# Usually you set "language" from the command line for these cases.
language = None
language = "en"

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
Expand Down Expand Up @@ -305,23 +300,6 @@
# If true, `todo` and `todoList` produce output, else they produce nothing.
todo_include_todos = False

# -- Extension configuration -------------------------------------------------

# nbsphinx_prolog = """
# see here for prolog/epilog details:
# https://nbsphinx.readthedocs.io/en/0.3.1/prolog-and-epilog.html
# """

nbsphinx_epilog = """
.. note::
Found a bug, or didn't find what you were looking for? 🙏 `Please file a
ticket <https://github.com/kedro-org/kedro/issues/new/choose>`_
"""

# -- NBconvert kedro config -------------------------------------------------
nbsphinx_kedro_name = "kedro"

# -- Kedro specific configuration -----------------------------------------
KEDRO_MODULES = [
"kedro.io",
Expand Down Expand Up @@ -493,20 +471,6 @@ def autodoc_process_docstring(app, what, name, obj, options, lines):
remove_arrows_in_examples(lines)


def _prepare_build_dir(app, config):
"""Get current working directory to the state expected
by the ReadTheDocs builder. Shortly, it does the same as
./build-docs.sh script except not running `sphinx-build` step."""
build_root = Path(app.srcdir)
build_out = Path(app.outdir)
copy_tree(str(here / "source"), str(build_root))
copy_tree(str(build_root / "api_docs"), str(build_root))
shutil.rmtree(str(build_root / "api_docs"))
shutil.rmtree(str(build_out), ignore_errors=True)
copy_tree(str(build_root / "css"), str(build_out / "_static" / "css"))
shutil.rmtree(str(build_root / "css"))


def env_override(default_appid):
build_version = os.getenv("READTHEDOCS_VERSION")

Expand All @@ -533,7 +497,6 @@ def _add_jinja_filters(app):


def setup(app):
app.connect("config-inited", _prepare_build_dir)
app.connect("builder-inited", _add_jinja_filters)
app.connect("autodoc-process-docstring", autodoc_process_docstring)
app.add_css_file("css/qb1-sphinx-rtd.css")
Expand Down Expand Up @@ -573,4 +536,4 @@ def setup(app):
# https://github.com/mermaidjs/mermaid.cli#linux-sandbox-issue
mermaid_params = ["-p", here / "puppeteer-config.json", "-s", "2"]
# https://github.com/kedro-org/kedro/issues/2451
mermaid_version = mermaid_init_js = None
mermaid_version = mermaid_init_js = ""
2 changes: 1 addition & 1 deletion docs/source/contribution/development_for_databricks.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
This guide describes how to efficiently develop features and fixes for Kedro on Databricks. Using this guide, you will be able to quickly test your locally modified version of Kedro on Databricks as part of a build-and-test development cycle.

```{note}
This page is for people developing changes to Kedro that need to test them on Databricks. If you are working on a Kedro project and need more information about project-deployment, consult the [documentation for deploying Kedro projects on Databricks](../deployment/databricks.md).
This page is for people developing changes to Kedro that need to test them on Databricks. If you are working on a Kedro project and need more information about workflows, consult the [documentation for developing a Kedro project on Databricks](../integrations/databricks_workspace.md).
```

## Prerequisites
Expand Down
21 changes: 1 addition & 20 deletions docs/source/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -186,30 +186,11 @@ Welcome to Kedro's documentation!
deployment/airflow_astronomer
deployment/dask

.. toctree::
:maxdepth: 2
:caption: Databricks integration

databricks_integration/visualisation

.. toctree::
:maxdepth: 2
:caption: PySpark integration

tools_integration/pyspark

.. toctree::
:maxdepth: 2
:caption: FAQs

faq/faq
faq/architecture_overview
faq/kedro_principles

.. toctree::
:maxdepth: 2
:caption: Resources

faq/faq
resources/glossary


Expand Down
2 changes: 1 addition & 1 deletion docs/source/integrations/databricks.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,5 @@
:maxdepth: 2
databricks_workspace.md
visualisation.md
databricks_visualisation.md
```
8 changes: 8 additions & 0 deletions docs/source/integrations/pyspark.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# PySpark integration

```{toctree}
:caption: PySpark
:maxdepth: 2
pyspark_integration.md
```
9 changes: 0 additions & 9 deletions docs/source/integrations/pyspark.rst

This file was deleted.

File renamed without changes.
File renamed without changes.
File renamed without changes.
10 changes: 10 additions & 0 deletions docs/source/kedro.framework.cli.jupyter.JupyterCommandGroup.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
kedro.framework.cli.jupyter.JupyterCommandGroup
===============================================

.. currentmodule:: kedro.framework.cli.jupyter

.. autoclass:: JupyterCommandGroup
:members:

.. Removed all methods and properties,
.. see https://github.com/kedro-org/kedro/issues/2453
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes
4 changes: 2 additions & 2 deletions docs/source/kedro_project_setup/starters.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,8 +49,8 @@ The Kedro team maintains the following starters for a range of Kedro projects:
* [`astro-airflow-iris`](https://github.com/kedro-org/kedro-starters/tree/main/astro-airflow-iris): The [Kedro Iris dataset example project](../get_started/new_project.md) with a minimal setup for deploying the pipeline on Airflow with [Astronomer](https://www.astronomer.io/).
* [`standalone-datacatalog`](https://github.com/kedro-org/kedro-starters/tree/main/standalone-datacatalog): A minimum setup to use the traditional [Iris dataset](https://www.kaggle.com/uciml/iris) with Kedro's [`DataCatalog`](../data/data_catalog.md), which is a core component of Kedro. This starter is of use in the exploratory phase of a project. For more information, read the guide to [standalone use of the `DataCatalog`](../notebooks_and_ipython/kedro_and_notebooks.md). This starter was formerly known as `mini-kedro`.
* [`pandas-iris`](https://github.com/kedro-org/kedro-starters/tree/main/pandas-iris): The [Kedro Iris dataset example project](../get_started/new_project.md)
* [`pyspark-iris`](https://github.com/kedro-org/kedro-starters/tree/main/pyspark-iris): An alternative Kedro Iris dataset example, using [PySpark](../tools_integration/pyspark.md)
* [`pyspark`](https://github.com/kedro-org/kedro-starters/tree/main/pyspark): The configuration and initialisation code for a [Kedro pipeline using PySpark](../tools_integration/pyspark.md)
* [`pyspark-iris`](https://github.com/kedro-org/kedro-starters/tree/main/pyspark-iris): An alternative Kedro Iris dataset example, using [PySpark](../integrations/pyspark_integration.md)
* [`pyspark`](https://github.com/kedro-org/kedro-starters/tree/main/pyspark): The configuration and initialisation code for a [Kedro pipeline using PySpark](../integrations/pyspark_integration.md)
* [`spaceflights`](https://github.com/kedro-org/kedro-starters/tree/main/spaceflights): The [spaceflights tutorial](../tutorial/spaceflights_tutorial.md) example code

## Starter versioning
Expand Down
2 changes: 1 addition & 1 deletion docs/source/nodes_and_pipelines/run_a_pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ kedro run --runner=ThreadRunner
`SparkDataSet` doesn't work correctly with `ParallelRunner`. To add concurrency to the pipeline with `SparkDataSet`, you must use `ThreadRunner`.
```

For more information on how to maximise concurrency when using Kedro with PySpark, please visit our guide on [how to build a Kedro pipeline with PySpark](../tools_integration/pyspark.md).
For more information on how to maximise concurrency when using Kedro with PySpark, please visit our guide on [how to build a Kedro pipeline with PySpark](../integrations/pyspark_integration.md).

hook_manager: PluginManager = None,

Expand Down
File renamed without changes.
File renamed without changes.
2 changes: 1 addition & 1 deletion docs/source/tutorial/add_another_pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -518,7 +518,7 @@ kedro run --runner=ThreadRunner
kedro run --runner=module.path.to.my.runner
```

`ParallelRunner` performs task parallelisation via multiprocessing, while `ThreadRunner` is intended for use with remote execution engines such as [Spark](../tools_integration/pyspark.md) and [Dask](/kedro.datasets.dask.ParquetDataSet).
`ParallelRunner` performs task parallelisation via multiprocessing, while `ThreadRunner` is intended for use with remote execution engines such as [Spark](../integrations/pyspark_integration.md) and [Dask](/kedro.datasets.dask.ParquetDataSet).

You can find out more about the runners Kedro provides, and how to create your own, in the [pipeline documentation about runners](../nodes_and_pipelines/run_a_pipeline.md).
atasets to work with different data formats (including CSV, Excel, and Parquet)
2 changes: 1 addition & 1 deletion docs/source/tutorial/package_a_project.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,4 +151,4 @@ There are various methods to deploy packaged pipelines via Kedro plugins:

* [Kedro-Docker](https://github.com/kedro-org/kedro-plugins/tree/main/kedro-docker) plugin for packaging and shipping Kedro projects within [Docker](https://www.docker.com/) containers.
* [Kedro-Airflow](https://github.com/kedro-org/kedro-plugins/tree/main/kedro-airflow) to convert your Kedro project into an [Airflow](https://airflow.apache.org/) project.
* The [Deployment guide](../deployment/deployment_guide) touches on other deployment targets such as AWS Batch and Prefect, and there is a [range of third-party plugins for deployment](extend_kedro/plugins.md#community-developed-plugins).
* The [Deployment guide](../deployment/deployment_guide) touches on other deployment targets such as AWS Batch and Prefect, and there is a [range of third-party plugins for deployment](../extend_kedro/plugins.md#community-developed-plugins).
Loading

0 comments on commit 38dfe6f

Please sign in to comment.