feat(core): expose service and task dependency outputs at runtime (#1123

)

* feat(core): expose service and task dependency outputs at runtime

This is a fairly substantial addition, which allows users to reference
runtime outputs from services and tasks in dependant services and tasks.
It involves adding a couple of new TaskGraph tasks, so that we can
resolve service statuses and task results in dependency order. We also
substantially overhaul the usage and testing of runtime contexts across
the board, since it is now a more fundamental feature.

The usage is documented in a new section in the Configuration Files
guide, please start with that when reviewing.

docs/reference/module-types/container.md

@@ -13,8 +13,7 @@ guide](../../using-garden/configuration-files.md).
 The [first section](#configuration-keys) lists and describes the available
 schema keys. The [second section](#complete-yaml-schema) contains the complete YAML schema.
 
-`container` modules also export values that are available in template strings under `${modules.<module-name>.outputs}`.
-See the [Outputs](#outputs) section below for details.
+`container` modules also export values that are available in template strings. See the [Outputs](#outputs) section below for details.
 
 ## Configuration keys
 
@@ -1044,6 +1043,8 @@ tasks:
 
 ## Outputs
 
+### Module outputs
+
 The following keys are available via the `${modules.<module-name>}` template string key for `container`
 modules.
 
@@ -1132,3 +1133,18 @@ outputs:
   ...
   deployment-image-name: "my-deployment-registry.io/my-org/my-module"
 ```
+
+
+### Task outputs
+
+The following keys are available via the `${runtime.tasks.<task-name>}` template string key for `container` module tasks.
+Note that these are only resolved when deploying/running dependants of the task, so they are not usable for every field.
+
+### `runtime.tasks.<task-name>.outputs.log`
+
+The full log from the executed task. (Pro-tip: Make it machine readable so it can be parsed by dependant tasks and services!)
+
+| Type     | Required | Default |
+| -------- | -------- | ------- |
+| `string` | No       | `""`    |
+

docs/reference/module-types/exec.md

@@ -8,8 +8,7 @@ guide](../../using-garden/configuration-files.md).
 The [first section](#configuration-keys) lists and describes the available
 schema keys. The [second section](#complete-yaml-schema) contains the complete YAML schema.
 
-`exec` modules also export values that are available in template strings under `${modules.<module-name>.outputs}`.
-See the [Outputs](#outputs) section below for details.
+`exec` modules also export values that are available in template strings. See the [Outputs](#outputs) section below for details.
 
 ## Configuration keys
 
@@ -380,6 +379,8 @@ tests:
 
 ## Outputs
 
+### Module outputs
+
 The following keys are available via the `${modules.<module-name>}` template string key for `exec`
 modules.
 
@@ -432,3 +433,18 @@ The outputs defined by the module.
 | Type     | Required |
 | -------- | -------- |
 | `object` | Yes      |
+
+
+### Task outputs
+
+The following keys are available via the `${runtime.tasks.<task-name>}` template string key for `exec` module tasks.
+Note that these are only resolved when deploying/running dependants of the task, so they are not usable for every field.
+
+### `runtime.tasks.<task-name>.outputs.log`
+
+The full log from the executed task. (Pro-tip: Make it machine readable so it can be parsed by dependant tasks and services!)
+
+| Type     | Required | Default |
+| -------- | -------- | ------- |
+| `string` | No       | `""`    |
+

docs/reference/module-types/helm.md

@@ -8,8 +8,7 @@ guide](../../using-garden/configuration-files.md).
 The [first section](#configuration-keys) lists and describes the available
 schema keys. The [second section](#complete-yaml-schema) contains the complete YAML schema.
 
-`helm` modules also export values that are available in template strings under `${modules.<module-name>.outputs}`.
-See the [Outputs](#outputs) section below for details.
+`helm` modules also export values that are available in template strings. See the [Outputs](#outputs) section below for details.
 
 ## Configuration keys
 
@@ -789,6 +788,8 @@ valueFiles: []
 
 ## Outputs
 
+### Module outputs
+
 The following keys are available via the `${modules.<module-name>}` template string key for `helm`
 modules.
 
@@ -851,3 +852,4 @@ The Helm release name of the service.
 | Type     | Required |
 | -------- | -------- |
 | `string` | Yes      |
+

docs/reference/module-types/kubernetes.md

@@ -16,8 +16,7 @@ guide](../../using-garden/configuration-files.md).
 The [first section](#configuration-keys) lists and describes the available
 schema keys. The [second section](#complete-yaml-schema) contains the complete YAML schema.
 
-`kubernetes` modules also export values that are available in template strings under `${modules.<module-name>.outputs}`.
-See the [Outputs](#outputs) section below for details.
+`kubernetes` modules also export values that are available in template strings. See the [Outputs](#outputs) section below for details.
 
 ## Configuration keys
 
@@ -298,6 +297,8 @@ files: []
 
 ## Outputs
 
+### Module outputs
+
 The following keys are available via the `${modules.<module-name>}` template string key for `kubernetes`
 modules.
 
@@ -350,3 +351,4 @@ The outputs defined by the module.
 | Type     | Required |
 | -------- | -------- |
 | `object` | Yes      |
+

docs/reference/module-types/maven-container.md

@@ -18,8 +18,7 @@ guide](../../using-garden/configuration-files.md).
 The [first section](#configuration-keys) lists and describes the available
 schema keys. The [second section](#complete-yaml-schema) contains the complete YAML schema.
 
-`maven-container` modules also export values that are available in template strings under `${modules.<module-name>.outputs}`.
-See the [Outputs](#outputs) section below for details.
+`maven-container` modules also export values that are available in template strings. See the [Outputs](#outputs) section below for details.
 
 ## Configuration keys
 
@@ -1082,6 +1081,8 @@ mvnOpts: []
 
 ## Outputs
 
+### Module outputs
+
 The following keys are available via the `${modules.<module-name>}` template string key for `maven-container`
 modules.
 
@@ -1170,3 +1171,4 @@ outputs:
   ...
   deployment-image-name: "my-deployment-registry.io/my-org/my-module"
 ```
+

docs/reference/module-types/openfaas.md

@@ -8,8 +8,7 @@ guide](../../using-garden/configuration-files.md).
 The [first section](#configuration-keys) lists and describes the available
 schema keys. The [second section](#complete-yaml-schema) contains the complete YAML schema.
 
-`openfaas` modules also export values that are available in template strings under `${modules.<module-name>.outputs}`.
-See the [Outputs](#outputs) section below for details.
+`openfaas` modules also export values that are available in template strings. See the [Outputs](#outputs) section below for details.
 
 ## Configuration keys
 
@@ -330,6 +329,8 @@ tests:
 
 ## Outputs
 
+### Module outputs
+
 The following keys are available via the `${modules.<module-name>}` template string key for `openfaas`
 modules.
 
@@ -392,3 +393,4 @@ The full URL to query this service _from within_ the cluster.
 | Type     | Required |
 | -------- | -------- |
 | `string` | Yes      |
+

docs/reference/providers/kubernetes.md

@@ -935,8 +935,7 @@ providers:
 
 ## Outputs
 
-The following keys are available via the `${providers.<provider-name>}` template string key for `kubernetes`
-providers.
+The following keys are available via the `${providers.<provider-name>}` template string key for `kubernetes` providers.
 
 ### `providers.<provider-name>.app-namespace`
 

docs/reference/providers/local-kubernetes.md

@@ -839,8 +839,7 @@ providers:
 
 ## Outputs
 
-The following keys are available via the `${providers.<provider-name>}` template string key for `local-kubernetes`
-providers.
+The following keys are available via the `${providers.<provider-name>}` template string key for `local-kubernetes` providers.
 
 ### `providers.<provider-name>.app-namespace`
 

docs/reference/template-strings.md

@@ -200,6 +200,34 @@ providers: {}
 #
 modules: {}
 
+# Runtime outputs and information from services and tasks (only resolved at runtime when deploying
+# services and running tasks).
+#
+# Type: object
+#
+runtime:
+  # Runtime information from the services that the service/task being run depends on.
+  #
+  # Type: object
+  #
+  # Example:
+  #   my-service:
+  #     outputs:
+  #       some-key: some value
+  #
+  services: {}
+
+  # Runtime information from the tasks that the service/task being run depends on.
+  #
+  # Type: object
+  #
+  # Example:
+  #   my-task:
+  #     outputs:
+  #       some-key: some value
+  #
+  tasks: {}
+
 # A map of all variables defined in the project configuration.
 #
 # Type: object

docs/using-garden/configuration-files.md

@@ -457,6 +457,36 @@ services:
 For a full reference of the keys available in template strings, please look at the
 [Template Strings Reference](../reference/template-strings.md).
 
+#### Runtime outputs
+
+Template keys prefixed with `runtime.` have some special semantics. They are used to expose runtime outputs from services and tasks, and therefore are resolved later than other template strings. _This means that you cannot use them for some fields, such as most identifiers, because those need to be resolved before validating the configuration._
+
+That caveat aside, they can be very handy when passing information between services and tasks. For example, you can pass log outputs from one task to another:
+
+```yaml
+kind: Module
+type: exec
+name: module-a
+tasks:
+  - name: prep-task
+    command: [echo, "output from my preparation task"]
+---
+kind: Module
+type: container
+name: my-container
+services:
+  - name: my-service
+    dependencies: [task-a]
+    env:
+      PREP_TASK_OUTPUT: ${runtime.tasks.prep-task.outputs.log}
+```
+
+Here the output from `prep-task` is copied to an environment variable for `my-service`. _Note that you currently need to explicitly declare `task-a` as a dependency for this to work._
+
+For a practical use case, you might for example make a task that provisions some infrastructure or prepares some data, and then passes information about it to services.
+
+Different module types expose different output keys for their services and tasks. Please refer to the [module type reference docs](https://docs.garden.io/reference/module-types) for details.
+
 #### Conditionals
 
 You can use conditional expressions in template strings, using the `||` operator. For example: