feat: migrate to plugins.v1 with filters & actions

This is a very large refactoring which aims at making Tutor both more
extendable and more generic. Historically, the Tutor plugin system was
designed as an ad-hoc solution to allow developers to modify their own
Open edX platforms without having to fork Tutor. The plugin API was
simple, but limited, because of its ad-hoc nature. As a consequence,
there were many things that plugin developers could not do, such as
extending different parts of the CLI or adding custom template filters.

Here, we refactor the whole codebase to make use of a generic plugin
system. This system was inspired by the Wordpress plugin API and the
Open edX "hooks and filters" API. The various components are added to a
small core thanks to a set of actions and filters. Actions are callback
functions that can be triggered at different points of the application
lifecycle. Filters are functions that modify some data. Both actions and
filters are collectively named as "hooks". Hooks can optionally be
created within a certain context, which makes it easier to keep track of
which application created which callback.

This new hooks system allows us to provide a Python API that developers
can use to extend their applications. The API reference is added to the
documentation, along with a new plugin development tutorial.

The plugin v0 API remains supported for backward compatibility of
existing plugins.

Done:
- Do not load commands from plugins which are not enabled.
- Load enabled plugins once on start.
- Implement contexts for actions and filters, which allow us to keep track of
  the source of every hook.
- Migrate patches
- Migrate commands
- Migrate plugin detection
- Migrate templates_root
- Migrate config
- Migrate template environment globals and filters
- Migrate hooks to tasks
- Generate hook documentation
- Generate patch reference documentation
- Add the concept of action priority

Close #499.

CHANGELOG.md

@@ -4,6 +4,9 @@ Note: Breaking changes between versions are indicated by "💥".
 
 ## Unreleased
 
+- 💥[Improvement] Complete overhaul of the plugin extension mechanism. Tutor now has a hook-based Python API: actions can be triggered at different points of the application life cycle and data can be modified thanks to custom filters. The v0 plugin API is still supported, for backward compatibility, but plugin developers are encouraged to migrate their plugins to the new API. See the new plugin tutorial for more information.
+- [Improvement] Improved the output of `tutor plugins list`.
+
 ## v13.1.11 (2022-04-12)
 
 - [Security] Apply SAML security fix.

Makefile

@@ -28,17 +28,19 @@ push-pythonpackage: ## Push python package to pypi
 
 test: test-lint test-unit test-types test-format test-pythonpackage ## Run all tests by decreasing order of priority
 
+test-static: test-lint test-types test-format  ## Run only static tests
+
 test-format: ## Run code formatting tests
 	black --check --diff $(BLACK_OPTS)
 
 test-lint: ## Run code linting tests
-	pylint --errors-only --enable=unused-import,unused-argument --ignore=templates ${SRC_DIRS}
+	pylint --errors-only --enable=unused-import,unused-argument --ignore=templates --ignore=docs/_ext ${SRC_DIRS}
 
 test-unit: ## Run unit tests
 	python -m unittest discover tests
 
 test-types: ## Check type definitions
-	mypy --exclude=templates --ignore-missing-imports --strict tutor/ tests/
+	mypy --exclude=templates --ignore-missing-imports --strict ${SRC_DIRS}
 
 test-pythonpackage: build-pythonpackage ## Test that package can be uploaded to pypi
 	twine check dist/tutor-$(shell make version).tar.gz
@@ -49,6 +51,9 @@ test-k8s: ## Validate the k8s format with kubectl. Not part of the standard test
 format: ## Format code automatically
 	black $(BLACK_OPTS)
 
+isort: ##  Sort imports. This target is not mandatory because the output may be incompatible with black formatting. Provided for convenience purposes.
+	isort --skip=templates ${SRC_DIRS}
+
 bootstrap-dev: ## Install dev requirements
 	pip install .
 	pip install -r requirements/dev.txt

bin/main.py

@@ -1,25 +1,17 @@
 #!/usr/bin/env python3
-from tutor.plugins import OfficialPlugin
+from tutor import hooks
 from tutor.commands.cli import main
+from tutor.plugins.v0 import OfficialPlugin
+
+
+@hooks.Actions.CORE_READY.add()
+def _discover_official_plugins() -> None:
+    # Manually discover plugins: that's because entrypoint plugins are not properly
+    # detected within the binary bundle.
+    with hooks.Contexts.PLUGINS.enter():
+        OfficialPlugin.discover_all()
 
-# Manually install plugins (this is for creating the bundle)
-for plugin_name in [
-    "android",
-    "discovery",
-    "ecommerce",
-    "forum",
-    "license",
-    "mfe",
-    "minio",
-    "notes",
-    "richie",
-    "webui",
-    "xqueue",
-]:
-    try:
-        OfficialPlugin.load(plugin_name)
-    except ImportError:
-        pass
 
 if __name__ == "__main__":
+    # Call the regular main function, which will not detect any entrypoint plugin
     main()

docs/Makefile

@@ -11,4 +11,7 @@ browse:
 	sensible-browser _build/html/index.html
 
 watch: build browse
-	while true; do inotifywait -e modify *.rst */*.rst */*/*.rst ../*.rst conf.py; $(MAKE) build || true; done
+	while true; do $(MAKE) wait-for-change build || true; done
+
+wait-for-change:
+	inotifywait -e modify $(shell find . -name "*.rst") ../*.rst ../tutor/hooks/*.py conf.py

docs/_ext/tutordocs.py

@@ -0,0 +1,14 @@
+"""
+This module is heavily inspired by Django's djangodocs.py:
+https://github.com/django/django/blob/main/docs/_ext/djangodocs.py
+"""
+from sphinx.application import Sphinx
+
+
+def setup(app: Sphinx) -> None:
+    # https://www.sphinx-doc.org/en/master/extdev/appapi.html#sphinx.application.Sphinx.add_crossref_type
+    app.add_crossref_type(
+        directivename="patch",
+        rolename="patch",
+        indextemplate="pair: %s; patch",
+    )

docs/conf.py

@@ -27,6 +27,10 @@
 exclude_patterns = ["_build", "Thumbs.db", ".DS_Store"]
 pygments_style = None
 
+# Autodocumentation of modules
+extensions.append("sphinx.ext.autodoc")
+autodoc_typehints = "description"
+
 # -- Sphinx-Click configuration
 # https://sphinx-click.readthedocs.io/
 extensions.append("sphinx_click")
@@ -108,5 +112,10 @@ def youtube(
     ]
 
 
-youtube.content = True
-docutils.parsers.rst.directives.register_directive("youtube", youtube)
+# Tutor's own extension
+sys.path.append(os.path.join(os.path.dirname(__file__), "_ext"))
+extensions.append("tutordocs")
+
+
+setattr(youtube, "content", True)
+docutils.parsers.rst.directives.register_directive("youtube", youtube)  # type: ignore

docs/index.rst

@@ -22,9 +22,9 @@
    gettingstarted
    run
    configuration
-   plugins
-   reference
-   tutorials
+   plugins/index
+   reference/index
+   tutorials/index
    troubleshooting
    tutor
    faq
@@ -59,6 +59,6 @@ This work is licensed under the terms of the `GNU Affero General Public License
 
 The AGPL license covers the Tutor code, including the Dockerfiles, but not the content of the Docker images which can be downloaded from https://hub.docker.com. Software other than Tutor provided with the docker images retain their original license.
 
-The Tutor plugin system is licensed under the terms of the `Apache License, Version 2.0 <https://opensource.org/licenses/Apache-2.0>`__.
+The Tutor plugin and hooks system is licensed under the terms of the `Apache License, Version 2.0 <https://opensource.org/licenses/Apache-2.0>`__.
 
 © 2021 Tutor is a registered trademark of SASU NULI NULI. All Rights Reserved.

docs/intro.rst

@@ -70,6 +70,8 @@ Urls:
 
 The platform is reset every day at 9:00 AM, `Paris (France) time <https://time.is/Paris>`__, so feel free to try and break things as much as you want.
 
+.. _how_does_tutor_work:
+
 How does Tutor work?
 --------------------
 

docs/plugins.rst → docs/plugins/index.rst

@@ -12,12 +12,10 @@ Tutor comes with a plugin system that allows anyone to customise the deployment
     # 3) Reconfigure and restart the platform
     tutor local quickstart
 
-For simple changes, it may be extremely easy to create a Tutor plugin: even non-technical users may get started with :ref:`simple YAML plugins <plugins_yaml>`.
+For simple changes, it may be extremely easy to create a Tutor plugin: even non-technical users may get started with our :ref:`plugin_development_tutorial` tutorial.
 
-In the following, we learn how to use and create Tutor plugins.
-
-Commands
---------
+Plugin commands cheatsheet
+--------------------------
 
 List installed plugins::
 
@@ -32,19 +30,23 @@ After enabling or disabling a plugin, the environment should be re-generated wit
 
     tutor config save
 
+The full plugins CLI is described in the :ref:`reference documentation <cli_plugins>`.
+
 .. _existing_plugins:
 
 Existing plugins
 ----------------
 
 Officially-supported plugins are listed on the `Overhang.IO <https://overhang.io/tutor/plugins>`__ website.
 
-Plugin development
-------------------
+Legacy plugin v0 development
+----------------------------
+
+.. include:: v0/legacy.rst
 
 .. toctree::
    :maxdepth: 2
 
-   plugins/api
-   plugins/gettingstarted
-   plugins/examples
+   v0/api
+   v0/gettingstarted
+   v0/examples

docs/plugins/api.rst → docs/plugins/v0/api.rst

@@ -1,15 +1,17 @@
 Plugin API
 ==========
 
+.. include:: legacy.rst
+
 Plugins can affect the behaviour of Tutor at multiple levels. They can:
 
-* Add new settings or modify existing ones in the Tutor configuration (see :ref:`config <plugin_config>`).
-* Add new templates to the Tutor project environment or modify existing ones (see :ref:`patches <plugin_patches>`, :ref:`templates <plugin_templates>` and :ref:`hooks <plugin_hooks>`).
-* Add custom commands to the Tutor CLI (see :ref:`command <plugin_command>`).
+* Add new settings or modify existing ones in the Tutor configuration (see :ref:`config <v0_plugin_config>`).
+* Add new templates to the Tutor project environment or modify existing ones (see :ref:`patches <v0_plugin_patches>`, :ref:`templates <v0_plugin_templates>` and :ref:`hooks <v0_plugin_hooks>`).
+* Add custom commands to the Tutor CLI (see :ref:`command <v0_plugin_command>`).
 
 There exist two different APIs to create Tutor plugins: either with YAML files or Python packages. YAML files are more simple to create but are limited to just configuration and template patches.
 
-.. _plugin_config:
+.. _v0_plugin_config:
 
 config
 ~~~~~~
@@ -42,19 +44,12 @@ This configuration from the "myplugin" plugin will set the following values:
 - ``MYPLUGIN_DOCKER_IMAGE``: this value will by default not be stored in ``config.yml``, but ``tutor config printvalue MYPLUGIN_DOCKER_IMAGE`` will print ``username/imagename:latest``.
 - ``MASTER_PASSWORD`` will be set to ``h4cked``. Needless to say, plugin developers should avoid doing this.
 
-.. _plugin_patches:
+.. _v0_plugin_patches:
 
 patches
 ~~~~~~~
 
-Plugin patches affect the rendered environment templates. In many places the Tutor templates include calls to ``{{ patch("patchname") }}``. This grants plugin developers the possibility to modify the content of rendered templates. Plugins can add content in these places by adding values to the ``patches`` attribute.
-
-.. note::
-    The list of existing patches can be found by searching for `{{ patch(` strings in the Tutor source code::
-
-        git grep "{{ patch"
-
-    The list of patches can also be browsed online `on Github <https://github.com/search?utf8=✓&q={{+patch+repo%3Aoverhangio%2Ftutor+path%3A%2Ftutor%2Ftemplates&type=Code&ref=advsearch&l=&l= 8>`__.
+Plugin patches affect the rendered environment templates. In many places the Tutor templates include calls to ``{{ patch("patchname") }}``. This grants plugin developers the possibility to modify the content of rendered templates. Plugins can add content in these places by adding values to the ``patches`` attribute. See :ref:`patches` for the complete list available patches.
 
 Example::
 
@@ -70,7 +65,7 @@ This will add a Redis instance to the services run with ``tutor local`` commands
     One can use this to dynamically load a list of patch files from a folder.
 
 
-.. _plugin_hooks:
+.. _v0_plugin_hooks:
 
 hooks
 ~~~~~
@@ -141,7 +136,7 @@ or::
     tutor images pull all
     tutor images push all
 
-.. _plugin_templates:
+.. _v0_plugin_templates:
 
 templates
 ~~~~~~~~~
@@ -162,7 +157,7 @@ In Tutor, templates are `Jinja2 <https://jinja.palletsprojects.com/en/2.11.x/>`_
 * ``list_if``: In a list of ``(value, condition)`` tuples, return the list of ``value`` for which the ``condition`` is true.
 * ``long_to_base64``: Base-64 encode a long integer.
 * ``iter_values_named``: Yield the values of the configuration settings that match a certain pattern. Example: ``{% for value in iter_values_named(prefix="KEY", suffix="SUFFIX")%}...{% endfor %}``. By default, only non-empty values are yielded. To iterate also on empty values, pass the ``allow_empty=True`` argument.
-* ``patch``: See :ref:`patches <plugin_patches>`.
+* ``patch``: See :ref:`patches <v0_plugin_patches>`.
 * ``random_string``: Return a random string of the given length composed of ASCII letters and digits. Example: ``{{ 8|random_string }}``.
 * ``reverse_host``: Reverse a domain name (see `reference <https://en.wikipedia.org/wiki/Reverse_domain_name_notation>`__). Example: ``{{ "demo.myopenedx.com"|reverse_host }}`` is equal to "com.myopenedx.demo".
 * ``rsa_import_key``: Import a PEM-formatted RSA key and return the corresponding object.
@@ -178,7 +173,7 @@ When saving the environment, template files that are stored in a template root w
 * Binary files with the following extensions: .ico, .jpg, .png, .ttf
 * Files that are stored in a folder named "partials", or one of its subfolders.
 
-.. _plugin_command:
+.. _v0_plugin_command:
 
 command
 ~~~~~~~

docs/plugins/examples.rst → docs/plugins/v0/examples.rst

@@ -3,6 +3,8 @@
 Examples of Tutor plugins
 =========================
 
+.. include:: legacy.rst
+
 The following are simple examples of :ref:`Tutor plugins <plugins>` that can be used to modify the behaviour of Open edX.
 
 Skip email validation for new users

docs/plugins/gettingstarted.rst → docs/plugins/v0/gettingstarted.rst

@@ -1,6 +1,8 @@
 Getting started with plugin development
 =======================================
 
+.. include:: legacy.rst
+
 Plugins can be created in two different ways: either as plain YAML files or installable Python packages. YAML files are great when you need to make minor changes to the default platform, such as modifying settings. For creating more complex applications, it is recommended to create python packages.
 
 .. _plugins_yaml:

docs/plugins/v0/legacy.rst

@@ -0,0 +1 @@
+.. warning:: The v0 plugin API is no longer the recommended way of developing new plugins for Tutor, starting from Tutor v13.2.0. See our :ref:`plugin creation tutorial <plugin_development_tutorial>` to learn more about the v1alpha plugin API. Existing v0 plugins will remain supported for some time but developers are encouraged to start migrating their plugins as soon as possible to make use of the new API.

docs/reference/api/hooks/actions.rst

@@ -0,0 +1,17 @@
+.. _actions:
+
+=======
+Actions
+=======
+
+Actions are one of the two types of hooks (the other being :ref:`filters`) that can be used to extend Tutor. Each action represents an event that can occur during the application life cycle. Each action has a name, and callback functions can be attached to it. When an action is triggered, these callback functions are called in sequence. Each callback function can trigger side effects, independently from one another.
+
+.. autofunction:: tutor.hooks.actions::get
+.. autofunction:: tutor.hooks.actions::get_template
+.. autofunction:: tutor.hooks.actions::add
+.. autofunction:: tutor.hooks.actions::do
+.. autofunction:: tutor.hooks.actions::clear
+.. autofunction:: tutor.hooks.actions::clear_all
+
+.. autoclass:: tutor.hooks.actions.Action
+.. autoclass:: tutor.hooks.actions.ActionTemplate

docs/reference/api/hooks/consts.rst

@@ -0,0 +1,23 @@
+=========
+Constants
+=========
+
+Here we lists named :ref:`actions`, :ref:`filters` and :ref:`contexts` that are used across Tutor. These are simply hook variables that we can refer to across the Tutor codebase without having to hard-code string names. The API is slightly different and less verbose than "native" hooks.
+
+Actions
+=======
+
+.. autoclass:: tutor.hooks.Actions
+    :members:
+
+Filters
+=======
+
+.. autoclass:: tutor.hooks.Filters
+    :members:
+
+Contexts
+========
+
+.. autoclass:: tutor.hooks.Contexts
+    :members:

docs/reference/api/hooks/contexts.rst

@@ -0,0 +1,11 @@
+.. _contexts:
+
+========
+Contexts
+========
+
+Contexts are a feature of the hook-based extension system in Tutor, which allows us to keep track of which components of the code created which callbacks. Contexts are very much an internal concept that most plugin developers should not have to worry about.
+
+.. autofunction:: tutor.hooks.contexts::enter
+
+.. autoclass:: tutor.hooks.contexts.Context

docs/reference/api/hooks/filters.rst

@@ -0,0 +1,20 @@
+.. _filters:
+
+=======
+Filters
+=======
+
+Filters are one of the two types of hooks (the other being :ref:`actions`) that can be used to extend Tutor. Filters allow one to modify the application behavior by transforming data. Each filter has a name, and callback functions can be attached to it. When a filter is applied, these callback functions are called in sequence; the result of each callback function is passed as the first argument to the next callback function. The result of the final callback function is returned to the application as the filter's output.
+
+.. autofunction:: tutor.hooks.filters::get
+.. autofunction:: tutor.hooks.filters::get_template
+.. autofunction:: tutor.hooks.filters::add
+.. autofunction:: tutor.hooks.filters::add_item
+.. autofunction:: tutor.hooks.filters::add_items
+.. autofunction:: tutor.hooks.filters::apply
+.. autofunction:: tutor.hooks.filters::iterate
+.. autofunction:: tutor.hooks.filters::clear
+.. autofunction:: tutor.hooks.filters::clear_all
+
+.. autoclass:: tutor.hooks.filters.Filter
+.. autoclass:: tutor.hooks.filters.FilterTemplate