From 545b4691422d61844b436343723e3b2fa20c3f1b Mon Sep 17 00:00:00 2001 From: Jon Edvald Date: Tue, 10 Sep 2019 14:43:15 +0200 Subject: [PATCH] docs: add guide for variables and templating --- docs/README.md | 1 + docs/reference/template-strings.md | 4 +- docs/using-garden/configuration-files.md | 205 +------------- docs/using-garden/variables-and-templating.md | 249 ++++++++++++++++++ .../src/docs/templates/template-strings.hbs | 4 +- 5 files changed, 261 insertions(+), 202 deletions(-) create mode 100644 docs/using-garden/variables-and-templating.md diff --git a/docs/README.md b/docs/README.md index 3164024e48..a2993c7690 100644 --- a/docs/README.md +++ b/docs/README.md @@ -16,6 +16,7 @@ * [Remote Kubernetes](./using-garden/remote-kubernetes.md) * [Remote Sources](./using-garden/using-remote-sources.md) * [Terraform](./using-garden/terraform.md) + * [Variables and templating](./using-garden/variables-and-templating.md) * [Example Projects](./examples/README.md) * [Demo Project](./examples/demo-project.md) * [TLS Project](./examples/tls-project.md) diff --git a/docs/reference/template-strings.md b/docs/reference/template-strings.md index 8ce9cd4e7b..0482b05ef7 100644 --- a/docs/reference/template-strings.md +++ b/docs/reference/template-strings.md @@ -7,7 +7,9 @@ Note that there are three sections below, since Project configs and Module confi them, and since additional keys are available under `providers` in Project configs. Please make sure to refer to the correct section. -Modules can reference `outputs` defined by other modules, via the `${modules..outputs}` key. +Modules can reference `outputs` defined by other modules, via the `${modules..outputs}` key, as well +as service and task outputs via the `${runtime.services..outputs}` and +`${runtime.tasks..outputs}`. For details on which outputs are available for a given module type, please refer to the [reference](https://docs.garden.io/reference/module-types) docs for the module type in question, and look for the _Outputs_ section. diff --git a/docs/using-garden/configuration-files.md b/docs/using-garden/configuration-files.md index e90b9179ca..ef8f0e652c 100644 --- a/docs/using-garden/configuration-files.md +++ b/docs/using-garden/configuration-files.md @@ -399,207 +399,12 @@ name: my-project dotIgnoreFiles: [.gardenignore] ``` -### Template strings - -String configuration values in `garden.yml` can be templated to inject variables, -information about the user's environment, references to other modules/services etc. - -The syntax for templated strings is `${some.key}`. The key is looked up from the context available when -resolving the string. The context depends on which top-level key the configuration value belongs to (`project` -or `module`). - -For example, for one service you might want to reference something from another module and expose it as an -environment variable: - -```yaml -kind: Module -name: some-module -services: - - name: some-service - ... - env: - OTHER_MODULE_VERSION: ${modules.other-module.version} -``` - -You can also inject a template variable into a string. For instance, you might need to include a module's -version as part of a URI: - -```yaml - ... - env: - OTHER_MODULE_ENDPOINT: http://other-module/api/${modules.other-module.version} -``` - -Note that while this syntax looks similar to template strings in Javascript, currently, only simple lookups by key -and conditionals are supported, whereas arbitrary JS expressions are not. - -Another common use case is to define `variables` in the project/environment configuration, and to use template strings -to propagate values to modules in the project: - -```yaml -kind: Project -... -variables: - log-level: "info" - ---- - -kind: Module -... -services: - - name: my-service - ... - env: - LOG_LEVEL: ${var.log-level} -``` - -For a full reference of the keys available in template strings, please look at the -[Template Strings Reference](../reference/template-strings.md). - -#### Runtime outputs - -Template keys prefixed with `runtime.` have some special semantics. They are used to expose runtime outputs from services and tasks, and therefore are resolved later than other template strings. _This means that you cannot use them for some fields, such as most identifiers, because those need to be resolved before validating the configuration._ - -That caveat aside, they can be very handy when passing information between services and tasks. For example, you can pass log outputs from one task to another: - -```yaml -kind: Module -type: exec -name: module-a -tasks: - - name: prep-task - command: [echo, "output from my preparation task"] ---- -kind: Module -type: container -name: my-container -services: - - name: my-service - dependencies: [task-a] - env: - PREP_TASK_OUTPUT: ${runtime.tasks.prep-task.outputs.log} -``` - -Here the output from `prep-task` is copied to an environment variable for `my-service`. _Note that you currently need to explicitly declare `task-a` as a dependency for this to work._ - -For a practical use case, you might for example make a task that provisions some infrastructure or prepares some data, and then passes information about it to services. - -Different module types expose different output keys for their services and tasks. Please refer to the [module type reference docs](https://docs.garden.io/reference/module-types) for details. - -#### Operators - -You can use a variety of operators in template string expressions: - -* Arithmetic: `*`, `/`, `%`, `+`, `-` -* Numeric comparison: `>=`, `<=`, `>`, `<` -* Equality: `==`, `!=` -* Logical: `&&`, `||`, ternary (` ? : `) -* Unary: `!` (negation), `typeof` (returns the type of the following value as a string, e.g. `"boolean"` or `"number"`) - -The arithmetic and numeric comparison operators can only be used for numeric literals and keys that resolve to numbers. The equality and logical operators work with any term. - -Clauses are evaluated in standard precedence order, but you can also use parentheses to control evaluation order (e.g. `${(1 + 2) * (3 + 4)}` evaluates to 21). - -These operators can be very handy, and allow you to tailor your configuration depending on different environments and other contextual variables. - -Here are some examples of usage: - -The `||` operator allows you to set default values: - -```yaml - # ... - variables: - log-level: ${local.env.LOG_LEVEL || "info"} - namespace: ${local.env.CI_BRANCH || local.username || "default"} -``` - -The `==` and `!=` operators allow you to set boolean flags based on other variables: - -```yaml -kind: Module -... -skipDeploy: ${environment.name == 'prod'} -... -``` - -```yaml -kind: Module -... -allowPublish: ${environment.name != 'prod'} -... -``` - -Ternary expressions, combined with comparison operators, can be good when provisioning resources: - -```yaml -kind: Module -type: container -... -services: - replicas: "${environment.name == 'prod' ? 3 : 1}" - ... -``` - -And the arithmetic operators can also be handy when provisioning resources: - -```yaml -kind: Module -type: container -... -services: - replicas: ${var.default-replicas * 2} - ... -``` - -#### Numbers, booleans and null values - -When a template string resolves to a number, boolean or null, its output is handled in one of two different ways, -depending on whether the template string is part of a surrounding string or not. - -If the template string is the whole string being interpolated, we assign the number, boolean or null directly to the -key: - -```yaml -kind: Project -... -variables: - default-replicas: 3 ---- -kind: Module -... -services: - - name: my-service - ... - replicas: ${var.default-replicas} # <- resolves to a number, as opposed to the string "3" -``` - -If, however, the template string is not the whole string being interpolated, but a component of it, the value is -formatted into the string, as you would expect: - -```yaml -kind: Project -... -variables: - project-id: 123 - some-key: null ---- -kind: Module -... -services: - - name: my-service - ... - env: - CONTEXT: project-${project-id} # <- resolves to "project-123" - SOME_VAR: foo-${var.some-key} # <- resolves to "foo-null" -``` - ## Next steps -We highly recommend browsing through the [Example projects](../examples/README.md) to see different examples of how projects and modules can be configured. +We highly recommend reading the [Variables and Templating guide](./variables-and-templating.md) to understand how you can reference across different providers and modules, as well as to understand how to supply secret values to your configuration. + +We suggest browsing through the [Example projects](../examples/README.md) as well, to see different examples of how projects and modules can be configured. -Also, be sure to look at the [Config Files Reference](../reference/config.md) for more details on each of the available -configuration fields, and the [Template Strings Reference](../reference/template-strings.md) for the keys available in -template strings. +Also, be sure to look at the [Config Files Reference](../reference/config.md) for more details on each of the available configuration fields, and the [Template Strings Reference](../reference/template-strings.md) for the keys available in template strings. -For deep-dives into specific use cases, you may want to look at the [Hot reload](./hot-reload.md) and -[Using Helm charts](./using-helm-charts.md) guides. +For deep-dives into specific use cases, you may want to look at the [Hot reload](./hot-reload.md) and [Using Helm charts](./using-helm-charts.md) guides. diff --git a/docs/using-garden/variables-and-templating.md b/docs/using-garden/variables-and-templating.md new file mode 100644 index 0000000000..a99f69dab8 --- /dev/null +++ b/docs/using-garden/variables-and-templating.md @@ -0,0 +1,249 @@ +# Variables and templating + +This guide introduces the templating capabilities available in Garden configuration files, the available ways to provide variable values, and how to reference outputs across modules and providers. + +## Template string basics + +String configuration values in `garden.yml` can be templated to inject variables, information about the user's environment, references to other modules/services and more. + +The syntax for templated strings is `${some.key}`. The key is looked up from the context available when resolving the string. The context depends on which top-level key the configuration value belongs to (`project` or `module`). + +For example, for one service you might want to reference something from another module and expose it as an environment variable: + +```yaml +kind: Module +name: some-module +services: + - name: some-service + ... + env: + OTHER_MODULE_VERSION: ${modules.other-module.version} +``` + +You can also inject a template variable into a string. For instance, you might need to include a module's +version as part of a URI: + +```yaml + ... + env: + OTHER_MODULE_ENDPOINT: http://other-module/api/${modules.other-module.version} +``` + +Note that while this syntax looks similar to template strings in Javascript, we don't allow arbitrary JS expressions. See the next section for the available expression syntax. + +### Operators + +You can use a variety of operators in template string expressions: + +* Arithmetic: `*`, `/`, `%`, `+`, `-` +* Numeric comparison: `>=`, `<=`, `>`, `<` +* Equality: `==`, `!=` +* Logical: `&&`, `||`, ternary (` ? : `) +* Unary: `!` (negation), `typeof` (returns the type of the following value as a string, e.g. `"boolean"` or `"number"`) + +The arithmetic and numeric comparison operators can only be used for numeric literals and keys that resolve to numbers. The equality and logical operators work with any term. + +Clauses are evaluated in standard precedence order, but you can also use parentheses to control evaluation order (e.g. `${(1 + 2) * (3 + 4)}` evaluates to 21). + +These operators can be very handy, and allow you to tailor your configuration depending on different environments and other contextual variables. + +Below are some examples of usage: + +The `||` operator allows you to set default values: + +```yaml + # ... + variables: + log-level: ${local.env.LOG_LEVEL || "info"} + namespace: ${local.env.CI_BRANCH || local.username || "default"} +``` + +The `==` and `!=` operators allow you to set boolean flags based on other variables: + +```yaml +kind: Module +... +skipDeploy: ${environment.name == 'prod'} +... +``` + +```yaml +kind: Module +... +allowPublish: ${environment.name != 'prod'} +... +``` + +Ternary expressions, combined with comparison operators, can be useful when provisioning resources: + +```yaml +kind: Module +type: container +... +services: + replicas: "${environment.name == 'prod' ? 3 : 1}" + ... +``` + +And the arithmetic operators can be handy when provisioning resources: + +```yaml +kind: Module +type: container +... +services: + replicas: ${var.default-replicas * 2} + ... +``` + +## Project variables + +A common use case for templating is to define variables in the project/environment configuration, and to use template strings to propagate values to modules in the project. + +You can define them in your project configuration using the [`variables` key](../reference/config.md#variables), as well as the [`environment[].variables` key](../reference/config.md#environment-variables) for environment-specific values. You might, for example, define project defaults using the `variables` key, and then provide environment-specific overrides in the `environment[].variables` key for each environment. + +The variables can then be configured via `${var.}` template string keys. For example: + +```yaml +kind: Project +... +variables: + log-level: info +environments: + - name: local + ... + variables: + log-level: debug + - name: remote + ... +--- +kind: Module +... +services: + - name: my-service + ... + env: + LOG_LEVEL: ${var.log-level} # <- resolves to "debug" for the "local" environment, "info" for the "remote" env +``` + +### Variable files (varfiles) + +You can also provide variables using "variable files" or _varfiles_. These work mostly like "dotenv" files or envfiles. However, they don't implicitly affect the environment of the Garden process and the configured services, but rather are added on top of the `variables` you define in your project `garden.yml`. + +This can be very useful when you need to provide secrets and other contextual values to your stack. You could add your varfiles to your `.gitignore` file to keep them out of your repository, or use e.g. [git-crypt](https://github.com/AGWA/git-crypt), [BlackBox](https://github.com/StackExchange/blackbox) or [git-secret](https://git-secret.io/) to securely store the files in your Git repo. + +By default, Garden will look for a `garden.env` file in your project root for project-wide variables, and a `garden..env` file for environment-specific variables. You can override the filename for each as well. The format of the files is the one supported by [dotenv](https://github.com/motdotla/dotenv#rules). + +The order of precedence across the varfiles and project config fields is as follows (from highest to lowest): +_`garden..env` file_ > _`environment[].variables` field_ > _`garden.env` file_ > _`variables` field + +Here's an example, where we have some project variables defined in our project config, and environment-specific values—including secret data—in varfiles: + +```yaml +# garden.yml +kind: Project +... +variables: + LOG_LEVEL: debug +environments: + - name: local + ... + - name: remote + ... +``` + +```plain +# garden.remote.env +log-level=info +database-password=fuin23liu54at90hiongl3g +``` + +```yaml +# my-service/garden.yml +kind: Module +... +services: + - name: my-service + ... + env: + LOG_LEVEL: ${var.log-level} + DATABASE_PASSWORD: ${var.database-password} +``` + +## Provider outputs + +Providers often expose useful variables that other provider configs and modules can reference, under `${providers..outputs.}`. Each provider exposes different outputs, and some providers have dynamic output keys depending on their configuration. + +For example, you may want to reference the app namespace from the [Kubernetes provider](../reference/providers/kubernetes.md) in module configs: + +```yaml +kind: Module +type: helm +... +values: + namespace: `${providers.kubernetes.outputs.app-namespace}` +``` + +Another good example is referencing outputs from Terraform stacks, via the [Terraform provider](./terraform.md): + +```yaml +kind: Module +type: container +services: + ... + env: + DATABASE_URL: `${providers.terraform.outputs.database_url}` # <- resolves the "database_url" stack output +``` + +Check out the individual [provider reference](../reference/providers/README.md) guides for details on what outputs each provider exposes. + +## Module outputs + +Modules often output useful information, that other modules can reference (provider configs cannot reference module outputs). Every module also exposes certain keys, like the module version. + +For example, you may want to reference the image name and version of a [container module](../reference/module-types/container.md): + +```yaml +kind: Module +type: helm +... +values: + # Resolves to the image name of the module, with the module version as the tag (e.g. "my-image:abcdef12345") + image: `${modules.my-image.outputs.deployment-image-name}:${modules.my-image.version}` +``` + +Check out the individual [module type reference](../reference/module-types/README.md) guides for details on what outputs each module type exposes. + +## Runtime outputs + +Template keys prefixed with `runtime.` have some special semantics. They are used to expose _runtime outputs_ from services and tasks, and therefore are resolved later than other template strings. _This means that you cannot use them for some fields, such as most identifiers, because those need to be resolved before validating the configuration._ + +That caveat aside, they can be very handy for passing information between services and tasks. For example, you can pass log outputs from one task to another: + +```yaml +kind: Module +type: exec +name: module-a +tasks: + - name: prep-task + command: [echo, "my task output"] +--- +kind: Module +type: container +name: my-container +services: + - name: my-service + dependencies: [task-a] + env: + PREP_TASK_OUTPUT: ${runtime.tasks.prep-task.outputs.log} # <- resolves to "my task output" +``` + +Here the output from `prep-task` is copied to an environment variable for `my-service`. _Note that you currently need to explicitly declare `task-a` as a dependency for this to work._ + +For a practical use case, you might for example make a task that provisions some infrastructure or prepares some data, and then passes information about it to services. + +Different module types expose different output keys for their services and tasks. Please refer to the [module type reference docs](https://docs.garden.io/reference/module-types) for details. + +## Next steps + +For a full reference of the keys available in template strings, please look at the [Template Strings Reference](../reference/template-strings.md), as well as individual [providers](../refrence/providers/README.md) for provider outputs, and [module types](../reference/module-types/README.md) for module and runtime output keys. diff --git a/garden-service/src/docs/templates/template-strings.hbs b/garden-service/src/docs/templates/template-strings.hbs index cd351a5419..90c852a18d 100644 --- a/garden-service/src/docs/templates/template-strings.hbs +++ b/garden-service/src/docs/templates/template-strings.hbs @@ -7,7 +7,9 @@ Note that there are three sections below, since Project configs and Module confi them, and since additional keys are available under `providers` in Project configs. Please make sure to refer to the correct section. -Modules can reference `outputs` defined by other modules, via the `${modules..outputs}` key. +Modules can reference `outputs` defined by other modules, via the `${modules..outputs}` key, as well +as service and task outputs via the `${runtime.services..outputs}` and +`${runtime.tasks..outputs}`. For details on which outputs are available for a given module type, please refer to the [reference](https://docs.garden.io/reference/module-types) docs for the module type in question, and look for the _Outputs_ section.