Extension callbacks refactoring and middleware implementation

docs/extensions/callbacks.md

@@ -5,7 +5,54 @@ Callback methods will get called from within Specter Desktop Core for various re
 
 Checkout the `cryptoadvance.specter.services.callbacks` file for all the specific callbacks.
 
+Your Extension is also able to specify your own callback-methods. What does that mean? Let's assume you have specified a callback `my_callback`. So anywhere in your code, once or as many times you like, you can call:
+```
+app.specter.service_manager.execute_ext_callbacks(my_callback, "any", params, you="like")
+```
+So the `service_manager` will then take care that all extensions are called which register that function. What exactly is called for each funtion and what's returned, depends on the `return_style` (see below). This is the same for all callbacks, no matter whether one which is called by core or by an extension.
+
 Some important one is the `after_serverpy_init_app` which passes a `Scheduler` class which can be used to setup regular tasks. A list of currently implemented callback-methods along with their descriptions are available in [`/src/cryptoadvance/specter/services/callbacks.py`](https://github.com/cryptoadvance/specter-desktop/blob/master/src/cryptoadvance/specter/services/callbacks.py).
 
+Those callback functions come in different `return_styles`: `collect` and `middleware`. 
+
+## Return Style `collect`
+
+Collect is the simpler style. In this model, all extensions are called exactly with the same arguments and the return-value is a dict whit the id's of the extensions as keys and the return-value as value.
+
+As an example, let's consider you specify this callback:
+```python
+# in service.py
+class MyExtension(Service):
+    callbacks = ["mynym.specterext.myextension.callbacks"]
+
+    def some_method(self):
+        returnvalues = app.specter.service_manager.execute_ext_callbacks(my_callback, "any", params, you="like")
+# in callbacks.py
+class my_callback(Callback)
+    id = "my_callback"
+```
+So now let's consider that there are two extensions which are speficied like this:
+```python
+
+class ExtensionA(Service):
+    id = "extensiona"
+    def callback_my_callback(any, params, you=like):
+        return { any:like }
+
+class ExtensionB(Service):
+    id = "extensionb"
+    def callback_my_callback(any, params, you=like):
+        return ["some","array"]
+```
+
+So in this case, the returnvalues would like this:
+```json
+{
+    "extensiona": {"any":"like"},
+    "extensionb": ["some","array"]
+}
+```
 
+## Return Style `middleware`
 
+In the case of middleware, you can pass one object which will in turn passed to all extensions which registered that callback. Have a look at the `adjust_view_model` callback which is explained in detail in the frontend-section.

docs/extensions/frontend-aspects.md

@@ -148,8 +148,6 @@ A reasonable `mywalletdetails.jinja` would look like this:
 
 ## Extending certain pages or complete endpoints (don't use this for now)
 
-Unfortunately this method is only able to be used once per Extension. So it's more or less unusable right now as there are already two extensions which are using those.
-
 For some endpoints, there is the possibility to extend/change parts of a page or the complete page. This works by declaring the `callback_adjust_view_model` method in your extension and modify the ViewModel which got passed into the callback. As there is only one callback for all types of ViewModels, you will need to check for the type that you're expecting and only adjust this type. Here is an example:
 
 ```python
@@ -163,10 +161,9 @@ class ExtensionidService(Service):
             # view_model.about_redirect=url_for("spectrum_endpoint.some_enpoint_here")
             # but we do it small here and only replace a specific component:
             view_model.get_started_include = "spectrum/welcome/components/get_started.jinja"
-            return view_model
-        return None
+        return view_model
 ```
-Make sure to return `None` if the `view_model` is not the type you're interested in.
+Make sure to return the view_model in anycase. No matter whether it's the correct type or not.
 
 In this example, a certain part of the page gets replaced. As you can read in the comments, you could also trigger a complete redirect to a different endpoint.
 

docs/extensions/intro.md

@@ -125,6 +125,8 @@ All the attributes of an extension are currently (json support is planned) defin
 Here is an example. This class definition MUST be stored in a file called "service.py" within a package with the name `mynym.specterext.myextensionid`. You don't have to create such files yourself. Please always use the extension generation procedure to create your extension.
 
 ```python
+from cryptoadvance.specterext.graphql.service.GraphqlService
+
 class MyextensionidService(Service):
     id = "myextensionid"
     name = "A Nice name for my Extension"
@@ -133,6 +135,8 @@ class MyextensionidService(Service):
     desc = "A nice description"
     has_blueprint = True
     blueprint_module = "mynym.specterext.myextensionid.controller"
+    callbacks = ["cryptoadvance.specterext.devhelp.callbacks"]
+    depends = [GraphqlService]
     isolated_client = False
     devices = ["mynym.specterext.myextensionid.devices.mydevice"]
     devstatus = devstatus_alpha
@@ -143,6 +147,11 @@ With inheriting from `Service` you get some useful methods explained later.
 The `id` needs to be unique within a specific Specter instance where this extension is part of. The `name` is the display name as shown to the user in the plugin-area (currently there is not yet a technical difference between extensions and plugins). The `icon` will be used where labels are used to be diplayed if this extension is reserving addresses. The `logo` and the `desc`ription is also used in the plugin area ("choose plugins").
 
 If the extension has a UI (currently all of them have one), `has_blueprint` is True. `The blueprint_module` is referencing the controller module where endpoints are defined. It's recommended to follow the format `org-id.specterext.myextensionid.controller`.
+
+As an extension, you can specify your own `callbacks`. Those are functions which can be registered by other extensions which can be called in your plugin-code whenever you think you want to provide an extension of functionality of your plugin. The attribute specifies the modules to search for those.
+
+Also you can let your extension depend on other extension. You should e.g. do that if you register callbacks which other extension has provided. In practice, it's determine the order of the callback functions which will be called. e.g. if your Extension A depends on a Extension B, and both extensions register a callback-function `adjust_view_model`, then the Extension B will get called before Extension A.
+
 `isolated_client` should not be used yet. It is determining where in the url-path tree the blueprint will be mounted. This might have an impact on whether the extension's frontend client has access to the cookie used in Specter. Check `config.py` for details.
 
 In `devices`, you can specify the modules where you're implementing new Devices.

pyinstaller/hooks/hook-cryptoadvance.specter.services.py

@@ -1,20 +1,20 @@
 import logging
-from cryptoadvance.specter.managers.service_manager import ServiceManager
+from cryptoadvance.specter.managers.service_manager import ExtensionManager
 
 logger = logging.getLogger(__name__)
 
 # Collecting template and static files from the different services in src/cryptoadvance/specter/services
 service_template_datas = [
     (service_dir, "templates")
-    for service_dir in ServiceManager.get_service_x_dirs("templates")
+    for service_dir in ExtensionManager.get_service_x_dirs("templates")
 ]
 service_static_datas = [
     (service_dir, "static")
-    for service_dir in ServiceManager.get_service_x_dirs("static")
+    for service_dir in ExtensionManager.get_service_x_dirs("static")
 ]
 
 # Collect Packages from the services, including service- and controller-classes
-service_packages = ServiceManager.get_service_packages()
+service_packages = ExtensionManager.get_service_packages()
 
 
 datas = [*service_template_datas, *service_static_datas]

requirements.in

@@ -28,10 +28,13 @@ gunicorn==20.1.0
 protobuf==3.20.2
 PyJWT==2.4.0
 pytimeparse==1.1.8
+# core-extension dependencies
+cryptoadvance.spectrum==0.3.1
 psycopg2-binary==2.9.5
+typing_extensions==4.0.0
+strawberry-graphql==0.155.2
 # Extensions
 cryptoadvance-liquidissuer==0.2.4
 specterext-exfund==0.1.7
 specterext-faucet==0.1.2
-cryptoadvance.spectrum==0.3.1
-specterext-stacktrack==0.2.1
+specterext-stacktrack==0.3.0

requirements.txt

@@ -678,9 +678,9 @@ specterext-exfund==0.1.7 \
 specterext-faucet==0.1.2 \
     --hash=sha256:86db78a6c41688152cfeec14efafd6d06c97e6edd9735461ee897495f90cb2e8
     # via -r requirements.in
-specterext-stacktrack==0.2.1 \
-    --hash=sha256:28729d1a981d8c061902b3d26baefc723d286aecea4d203ed729d1c89a9de267 \
-    --hash=sha256:34a5e9da8a3cb7a4c8b7bc4e0a417a043150e608a7261e56d84fbfba9af15b8a
+specterext-stacktrack==0.3.0 \
+    --hash=sha256:14f96f1f552f57ba017b8bc642f07343edbb1abafe09e03bbaae179d78d7ce23 \
+    --hash=sha256:9e2946185730aab377951e83a27d8791a34e0f031e44f15991212b6b85722ca0
     # via -r requirements.in
 sqlalchemy==1.4.42 \
     --hash=sha256:04f2598c70ea4a29b12d429a80fad3a5202d56dce19dd4916cc46a965a5ca2e9 \

src/cryptoadvance/specter/config.py

@@ -181,6 +181,7 @@ class BaseConfig(object):
         "cryptoadvance.specterext.electrum.service",
         "cryptoadvance.specterext.spectrum.service",
         "cryptoadvance.specterext.stacktrack.service",
+        # "cryptoadvance.specterext.graphql.service",
     ]
 
     # This is just a placeholder in order to be aware that you cannot set this

src/cryptoadvance/specter/managers/node_manager.py

@@ -11,7 +11,7 @@
 from ..node import Node
 from ..internal_node import InternalNode
 from ..services import callbacks
-from ..managers.service_manager import ServiceManager
+from ..managers.service_manager.service_manager import ExtensionManager
 from ..util.bitcoind_setup_tasks import setup_bitcoind_thread
 
 logger = logging.getLogger(__name__)
@@ -39,7 +39,7 @@ def __init__(
         self.only_tor = only_tor
         self.bitcoind_path = bitcoind_path
         self.internal_bitcoind_version = internal_bitcoind_version
-        self.service_manager: ServiceManager = service_manager
+        self.service_manager: ExtensionManager = service_manager
         self.load_from_disk(data_folder)
         internal_nodes = [
             node for node in self.nodes.values() if not node.external_node

src/cryptoadvance/specter/managers/service_manager/callback_executor.py

@@ -0,0 +1,142 @@
+import logging
+from typing import Dict, List
+
+from cryptoadvance.specter.util.reflection import (
+    get_template_static_folder,
+    get_subclasses,
+)
+from flask import current_app as app
+from flask import url_for
+from flask.blueprints import Blueprint
+
+from cryptoadvance.specter.util.specter_migrator import SpecterMigration
+
+from ...services.service import Service
+from ...services.callbacks import *
+from ...services.callbacks import Callback
+from ...specter_error import SpecterInternalException
+
+
+logger = logging.getLogger(__name__)
+
+
+class CallbackExecutor:
+    """encapsulating the complexities of the extension callbacks"""
+
+    def __init__(self, extensions):
+        self._extensions = extensions
+
+    def execute_ext_callbacks(self, callback: Callback, *args, **kwargs):
+        """will execute the callback function for each extension which has defined that method
+        the callback_id needs to be passed and specify why the callback has been called.
+        It needs to be one of the constants defined in cryptoadvance.specter.services.callbacks
+        """
+        self.check_callback(callback)
+        # No debug statement here possible as this is called for every request and would flood the logs
+        # logger.debug(f"Executing callback {callback_id}")
+        return_values = {}
+        for ext in self.services_sorted:
+            if hasattr(ext, f"callback_{callback.id}"):
+                # logger.debug(f"About to execute on ext {ext.id} callback_{callback.id}")
+                return_values[ext.id] = getattr(ext, f"callback_{callback.id}")(
+                    *args, **kwargs
+                )
+                # logger.debug(f"returned {return_values[ext.id]}")
+                if callback.return_style == "middleware":
+                    args = [return_values[ext.id]]
+        # Filtering out all None return values
+        return_values = {k: v for k, v in return_values.items() if v is not None}
+        # logger.debug(f"return_values for callback {callback.id} {return_values}")
+        if callback.return_style == "collect":
+            return return_values
+        elif callback.return_style == "middleware":
+            return args[0]
+        else:
+            raise SpecterInternalException(
+                f""" 
+                Unknown callback return_style {callback.return_style} for callback {callback}
+            """
+            )
+
+    @property
+    def extensions(self) -> Dict[str, Service]:
+        return self._extensions or {}
+
+    @property
+    def services_sorted(self) -> List[Service]:
+        """A list of sorted extensions. First sort-criteria is the dependency. Second one the sort-priority"""
+        if hasattr(self, "_services_sorted"):
+            return self._services_sorted
+        exts_sorted = topological_sort(self.extensions.values())
+        for ext in exts_sorted:
+            ext.__class__.dependency_level = 0
+        for ext in exts_sorted:
+            set_dependency_level_recursive(ext.__class__, 0)
+        exts_sorted.sort(
+            key=lambda x: (-x.dependency_level, -getattr(x, "sort_priority", 0))
+        )
+        self._services_sorted = exts_sorted
+        return self._services_sorted
+
+    @property
+    def all_callbacks(self):
+        return get_subclasses(Callback)
+
+    def check_callback(self, callback, *args, **kwargs):
+        if type(callback) != type:
+            callback: Callback = callback.__class__
+        if callback not in self.all_callbacks:
+            raise SpecterInternalException(
+                f"""
+                Non existing callback_id: {callback} or your class does not inherit from Callback
+                """
+            )
+        if callback.return_style == "middleware":
+            if len(args) > 1:
+                raise SpecterInternalException(
+                    f"""
+                The callback {callback} is using middleware but it's passing more than one argument: {args}."
+                """
+                )
+
+            if len(kwargs.values()) > 0:
+                raise SpecterInternalException(
+                    f"""
+                The callback {callback} is using middleware but it's using named arguments: {kwargs}.
+                """
+                )
+
+
+def topological_sort(instances):
+    """Sorts a list of instances so that non dependent ones come first"""
+    class_map = {instance.__class__: instance for instance in instances}
+    in_degree = {cls: 0 for cls in class_map.keys()}
+    graph = {cls: set() for cls in class_map.keys()}
+
+    for cls, instance in class_map.items():
+        for dep in getattr(cls, "depends", []):
+            graph[dep].add(cls)
+            in_degree[cls] += 1
+
+    no_incoming_edges = [cls for cls in class_map.keys() if in_degree[cls] == 0]
+    output = []
+
+    while no_incoming_edges:
+        node = no_incoming_edges.pop()
+        output.append(class_map[node])
+
+        for child in graph[node]:
+            in_degree[child] -= 1
+            if in_degree[child] == 0:
+                no_incoming_edges.append(child)
+
+    if len(output) != len(class_map):
+        raise ValueError("Graph contains a cycle.")
+
+    return output
+
+
+def set_dependency_level_recursive(ext, level=0):
+    for dep in getattr(ext, "depends", []):
+        dep.dependency_level = max(getattr(dep, "dependency_level", 0), level + 1)
+        set_dependency_level_recursive(dep, level + 1)