Add documentation regarding the transfer optimization feature. (Issue i…

…nmanta/inmanta-lsm#1794, PR #7922)

# Description

Add documentation about `transfer optimization` feature.

Closes inmanta/inmanta-lsm#1794

# Self Check:

- [x] Attached issue to pull request
- [x] Changelog entry
- [x] Type annotations are present
- [x] Code is clear and sufficiently documented
- [x] No (preventable) type errors (check using make mypy or make mypy-diff)
- [x] Sufficient test cases (reproduces the bug/tests the requested feature)
- [x] Correct, in line with design
- [x] End user documentation is included or an issue is created for end-user documentation
- [ ] ~~If this PR fixes a race condition in the test suite, also push the fix to the relevant stable branche(s) (see [test-fixes](https://internal.inmanta.com/development/core/tasks/build-master.html#test-fixes) for more info)~~

changelogs/unreleased/document-transfer-optimization.yml

@@ -0,0 +1,6 @@
+---
+description: Add documentation regarding the transfer optimization feature.
+issue-nr: 1794
+issue-repo: inmanta-lsm
+change-type: minor
+destination-branches: [master, iso7]

docs/lsm/index.rst

@@ -35,6 +35,7 @@ and change the refinement process based on external events.
     validation_types/*
     service_identity
     state_transfer_tx
+    transfer_optimization/*
 
 
 Service catalog

docs/lsm/transfer_optimization/sources/basic_lifecycle.cf

@@ -0,0 +1,145 @@
+import lsm
+import lsm::fsm
+import std::testing
+
+start = lsm::State(
+    name="start",
+    export_resources=false,
+    validate_self="candidate",
+)
+creating = lsm::State(
+    name="creating",
+    export_resources=true,
+    validate_self="active",
+    validate_others="active",
+)
+up = lsm::State(
+    name="up",
+    label="success",
+    export_resources=true,
+    validate_self="active",
+    validate_others="active",
+)
+failed = lsm::State(
+    name="failed",
+    label="danger",
+    export_resources=true,
+    validate_self="active",
+    validate_others="active",
+)
+rejected = lsm::State(
+    name="rejected",
+    label="danger",
+    export_resources=false,
+    deleted=true,
+)
+deleting = lsm::State(
+    name="deleting",
+    export_resources=true,
+    validate_self="active",
+    validate_others="active",
+    purge_resources=true,
+)
+terminated = lsm::State(
+    name="terminated",
+    export_resources=false,
+    deleted=true,
+)
+
+basic_lifecycle = lsm::LifecycleStateMachine(
+    name="testing::basic_lifecycle",
+    initial_state=start,
+    transfers=[
+        lsm::StateTransfer(
+            source=start,
+            target=creating,
+            error=rejected,
+            validate=true,
+            auto=true,
+            target_operation="promote",
+            error_same_desired_state=true,
+        ),
+        lsm::StateTransfer(
+            source=creating,
+            target=up,
+            error=creating,
+            resource_based=true,
+            target_same_desired_state=true,
+            error_same_desired_state=true,
+        ),
+        lsm::StateTransfer(
+            source=up,
+            target=up,
+            error=failed,
+            resource_based=true,
+            target_same_desired_state=true,
+            error_same_desired_state=true,
+        ),
+        lsm::StateTransfer(
+            source=failed,
+            target=up,
+            error=failed,
+            resource_based=true,
+            target_same_desired_state=true,
+            error_same_desired_state=true,
+        ),
+        lsm::StateTransfer(
+            source=up,
+            target=deleting,
+            on_delete=true,
+        ),
+        lsm::StateTransfer(
+            source=deleting,
+            target=terminated,
+            error=deleting,
+            resource_based=true,
+            error_same_desired_state=true,
+        ),
+    ]
+)
+
+entity InterfaceIPAssignment extends lsm::ServiceEntity:
+    """
+        Interface details.
+
+        :attr service_id: A unique ID for this service.
+
+        :attr router_ip: The IP address of the SR linux router that should be configured.
+        :attr router_name: The name of the SR linux router that should be configured.
+        :attr interface_name: The name of the interface of the router that should be configured.
+        :attr address: The IP-address to assign to the given interface.
+    """
+    string service_id
+
+    string router_ip
+    string router_name
+    string interface_name
+
+    string address
+    lsm::attribute_modifier address__modifier="rw+"
+
+end
+
+index InterfaceIPAssignment(service_id)
+
+implementation interface_config for InterfaceIPAssignment:
+    """ Add a dummy resource to the interface ip assignment to represent actual configuration """
+    self.resources += std::testing::NullResource(name=self.address)
+    self.owned_resources += self.resources # We own all our resources and nothing else
+end
+
+implement InterfaceIPAssignment using parents, interface_config
+
+binding = lsm::ServiceEntityBindingV2(
+    service_entity="__config__::InterfaceIPAssignment",
+    lifecycle=basic_lifecycle,
+    service_entity_name="ip_assignment",
+)
+
+for instance in lsm::all(binding):
+    InterfaceIPAssignment(
+        instance_id=instance["id"],
+        entity_binding=binding,
+        **instance["attributes"],
+    )
+end

docs/lsm/transfer_optimization/transfer_optimization.rst

@@ -0,0 +1,59 @@
+*********************
+Transfer optimization
+*********************
+
+By default, the Inmanta server performs a new compile every time the state of a service instance changes.
+However, in practice it often happens that a transition between two states doesn't result in a new desired state for
+the service instance. To prevent unnecessary compiles, the LSM module has support to indicate which
+transfers in a lifecycle preserve the desired state. The Inmanta server can then use this information to improve the
+performance of state transitions. This page describes how to mark transfers in a lifecycle as state-preserving
+and how to enable the transfer optimization feature on the server.
+
+Annotating desired state-preserving transfers
+=============================================
+
+The ``lsm::StateTransfer`` entity has two attributes to indicate that a transfer preserves the desired state:
+
+* `bool target_same_desired_state (default=false)`: True iff following the success edge doesn't change the desired state.
+* `bool error_same_desired_state (default=false)`: True iff following the error edge doesn't change the desired state.
+
+The code snippet below models a lifecycle that contains state-preserving transfers:
+
+.. literalinclude:: sources/basic_lifecycle.cf
+    :language: inmanta
+    :lines: 1-100
+    :linenos:
+
+Let's discuss the transfers that are not marked as state-preserving and why:
+
+* start -> creating: Not a state-preserving transfer because it moves the instances from a non-exporting
+  state to an exporting state.
+* up -> deleting: Not a state-preserving transfer because this state transfer flips the ``purge_resources`` flag,
+  which will have an effect on the desired state being deployed.
+* deleting -> terminated: Not a state-preserving transfer because it moves the instance from an exporting to a
+  non-exporting state.
+
+All other transfers were marked as state-preserving transfers. This decision was based on the assumption that not
+changing the high-level intent (active attribute set) doesn't change the low-level intent (what is deployed to the
+infrastructure). This assumption doesn't hold in all situations. The service model could for example change the
+low-level intent based on the current state of the service instance. Caution is advised when modelling a lifecycle
+with state-preserving transfer, as incorrectly marking a transfer as state-preserving will cause the orchestrator
+to behave incorrectly.
+
+Enabling the transfer optimization feature
+==========================================
+
+The environment setting :inmanta.environment-settings:setting:`enable_lsm_transfer_optimization` can be used to enable
+the transfer optimization feature. When enabled, the LSM extension will perform a compile only when transitioning between
+two states that don't preserve the desired state. When disabled, a recompile will be done for each state transition.
+Validation compiles are not impacted by this setting. They are always executed.
+
+Testing
+=======
+
+The Inmanta server validates every lifecycle that is exported to the server and it will reject any lifecycle that is
+indisputably wrong. The server will for example reject lifecycles with state-preserving transfers
+that connect a state that exports resources with a state that doesn't export resources. However, the server cannot
+exhaustively detect all cases where transfers are incorrectly marked as state-preserving. As such, it's important to
+add tests that validate whether the service model behaves consistently, whether or not the
+:inmanta.environment-settings:setting:`enable_lsm_transfer_optimization` environment setting is enabled.