Formalize evaluation state in controllers

[erwinvaneyk]

Previously, the controllers attempted to ensure exclusive access to an evaluation with some ad-hoc solutions. This PR improves this by introducing formalized constructs that deal with the evaluation state.

• EvalRecord captures all aspects (timestamps, results, logs...) of a single evaluation of the state of a workflow or invocation.
• EvalLog is an append-only log consisting out of EvalRecords
• EvalState is the current, complete evaluation state of a single invocation or workflow. This contains the logs, locks for exclusive access, and other metadata. Evaluations are provided with the EvalState of the associated workflow or invocation. A controller can, with an evalstate, gain exclusive access for evaluating the invocation or workflow.
• EvalCache is a data structure to store and retrieve EvalStates based on generic identifiers in a thread-safe manner. This removes some of the boilerplate otherwise needed by an controller to deal with evalstates.

Additionally, using this EvalState the controller logic has been restructured to a more consistent, simplified model. A controller in its essence (currently still contain a lot of plumbing that at some point can be extracted to a shared controller helper) consists solely out of evaluation(id string)to trigger the evaluation of an object with the relevant id. A controller will do the required fetching of object states, e.g. get the current and desired object status from cache or a backing store, evaluate the difference, and execute actions if necessary. The evaluation can be modeled as a set of rules, where each rule evaluates the state and returns the required action: rule : state -> action. In this PR this model has been implemented using the Rule and Action interfaces. The advantages of this approach include improved testability of the individual and combinations of rules and actions, reuse of rules across different controllers, and a clear separation of mechanism (controller, actions) and policy (rules).

Furthermore, as executing the actions typically does not introduce heavy load, the action workqueue has been removed. The exclusive access on the evaluation of an invocation or workflow is only removed after the action has been executed, preventing new evaluations before the previous action has been executed. Instead an evaluation queue has been added, with the controller ticker and notifications submitting evaluations to the queue. The controller evaluation loop uses this queue to determine the next id to evaluate, centralizing the entrypoint/usage of this control loop.

These changes together intend to make the controller approach more reliable, eliminate the existing (potential) concurrency issues, make the controllers better testable (as the added tests show) and provide a more simplified model of the controllers.

Additionally this PR:

• Improves/adds additional tests
• Implements the delete workflow endpoint on the apiserver
• Renames the Function events to Task events for consistency