diff --git a/docs/README.md b/docs/README.md index b3301357c5..9ff09fea1b 100644 --- a/docs/README.md +++ b/docs/README.md @@ -8,6 +8,7 @@ * [Features and usage](./using-garden/features-and-usage.md) * [Configuration files](./using-garden/configuration-files.md) * [Remote Clusters](./using-garden/remote-clusters.md) + * [Using Helm charts](./using-garden/using-helm-charts.md) * [Hot Reload](./using-garden/hot-reload.md) * [Example projects](./examples/README.md) * [Hello world](./examples/hello-world.md) diff --git a/docs/reference/config.md b/docs/reference/config.md index 2b23749648..49255101d1 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -46,15 +46,6 @@ module: # Optional. repositoryUrl: - # Variables that this module can reference and expose as environment variables. - # - # Example: - # my-variable: some-value - # - # Optional. - variables: - {} - # When false, disables pushing this module to remote registries. # # Optional. @@ -237,15 +228,6 @@ module: # Optional. repositoryUrl: - # Variables that this module can reference and expose as environment variables. - # - # Example: - # my-variable: some-value - # - # Optional. - variables: - {} - # When false, disables pushing this module to remote registries. # # Optional. @@ -409,15 +391,6 @@ module: # Optional. repositoryUrl: - # Variables that this module can reference and expose as environment variables. - # - # Example: - # my-variable: some-value - # - # Optional. - variables: - {} - # When false, disables pushing this module to remote registries. # # Optional. @@ -801,15 +774,6 @@ module: # Optional. repositoryUrl: - # Variables that this module can reference and expose as environment variables. - # - # Example: - # my-variable: some-value - # - # Optional. - variables: - {} - # When false, disables pushing this module to remote registries. # # Optional. @@ -858,7 +822,21 @@ module: # Optional. target: - # A valid Helm chart name or URI. Required if the module doesn't contain the Helm chart itself. + # The name of another `helm` module to use as a base for this one. Use this to re-use a Helm + # chart across multiple services. For example, you might have an organization-wide base chart + # for certain types of services. + # If set, this module will by default inherit the following properties from the base module: + # `serviceResource`, `values` + # Each of those can be overridden in this module. They will be merged with a JSON Merge Patch + # (RFC 7396). + # + # Example: "my-base-chart" + # + # Optional. + base: + + # A valid Helm chart name or URI (same as you'd input to `helm install`). Required if the module + # doesn't contain the Helm chart itself. # # Example: "stable/nginx-ingress" # @@ -866,7 +844,7 @@ module: chart: # The path, relative to the module path, to the chart sources (i.e. where the Chart.yaml file - # is, if any). + # is, if any). Not used when `base` is specified. # # Optional. chartPath: . @@ -942,6 +920,12 @@ module: hotReloadArgs: - + # Set this to true if the chart should only be built, but not deployed as a service. Use this, + # for example, if the chart should only be used as a base for other modules. + # + # Optional. + skipDeploy: false + # The task definitions for this module. # # Optional. diff --git a/docs/using-garden/using-helm-charts.md b/docs/using-garden/using-helm-charts.md index 0a5b06e31d..e980012b8c 100644 --- a/docs/using-garden/using-helm-charts.md +++ b/docs/using-garden/using-helm-charts.md @@ -151,3 +151,100 @@ module: For hot-reloading to work you must specify `serviceResource.containerModule`, so that Garden knows which module contains the sources to use for hot-reloading. You can then optionally add `serviceResource.hotReloadArgs` to, for example, start the container with automatic reloading or in development mode. For the above example, you could then run `garden deploy -w --hot-reload=vote` or `garden dev --hot-reload=vote` to start the `vote` service in hot-reloading mode. When you then change the sources in the _vote-image_ module, Garden syncs the changes to the running container from the Helm chart. + +## Re-using charts + +Often you'll want to re-use the same Helm charts for multiple modules. For example, you might have a generic template +for all your backend services that configures auto-scaling, secrets/keys, sidecars, routing and so forth, and you don't +want to repeat those configurations all over the place. + +You can achieve this by using the `base` field on the `helm` module type. Staying with our `vote-helm` example project, +let's look at the `base-chart` and `api` modules: + +```yaml +# base-chart +module: + description: Base Helm chart for services + type: helm + name: base-chart + serviceResource: + kind: Deployment + skipDeploy: true +``` + +```yaml +# api +module: + description: The API backend for the voting UI + type: helm + name: api + base: base-chart + serviceResource: + containerModule: api-image + dependencies: + - redis + values: + name: api + image: + repository: api-image + tag: ${modules.api-image.version} + ingress: + enabled: true + paths: [/] + hosts: [api.local.app.garden] +``` + +Here, the `base-chart` module contains the actual Helm chart and templates. Note the `skipDeploy` flag, which we set +because the module should only be used as a base chart in this case. + +The `api` module only contains the `garden.yml` file, but configures the base chart using the `values` field, and also +sets its own dependencies (those are not inherited) and specifies its `serviceResource.containerModule`. + +In our base chart, we make certain values like `name`, `image.repository` and `image.tag` required (using the +[required](https://github.com/helm/helm/blob/master/docs/charts_tips_and_tricks.md#using-the-required-function) +helper function) in order to enforce correct usage. We recommend enforcing constraints like that, so that mistakes +can be caught quickly. + +The `result` module also uses the same base chart, but sets different values and metadata: + +```yaml +module: + description: Helm chart for the results UI + type: helm + name: result + base: base-chart + serviceResource: + containerModule: result-image + hotReloadArgs: [nodemon, server.js] + dependencies: + - db-init + values: + name: result + image: + repository: result-image + tag: ${modules.result-image.version} + ingress: + enabled: true + paths: [/] + hosts: [result.local.app.garden] + tests: + - name: integ + args: [echo, ok] + dependencies: + - db-init +``` + +This pattern can be quite powerful, and can be used to share common templates across your organization. You could +even have an organization-wide repository of base charts for different purposes, and link it in your project config +with something like this: + +```yaml +project: + sources: + - name: base-charts + repositoryUrl: https://github.com/my-org/helm-base-charts.git#v0.1.0 + ... +``` + +The base chart can also be any `helm` module (not just "base" charts specifically made for that purpose), so you have +a lot of flexibility in how you organize your charts. diff --git a/examples/vote-helm/README.md b/examples/vote-helm/README.md index c37ace1b3a..eb85895b33 100644 --- a/examples/vote-helm/README.md +++ b/examples/vote-helm/README.md @@ -1,7 +1,7 @@ # Voting example project with Helm charts This is a clone of the [vote example project](../vote/README.md), modified to use Helm charts to describe -Kubernetes resources, instead of the `container` module type. +Kubernetes resources, instead of the simpler `container` module type. You'll notice that we still use the `container` module types for building the container images (the corresponding `*-image` next to each service module), but they do not contain a `service` section. @@ -9,5 +9,11 @@ You'll notice that we still use the `container` module types for building the co The `helm` modules only contain the charts, which reference the container images. Garden will build the images ahead of deploying the charts. +Furthermore, to showcase the chart re-use feature, the `api` and `result` modules use the `base-chart` module +as a base. + +For more details on how to use Helm charts, please refer to our +[Helm user guide](https://docs.garden.io/using-garden). + The usage and workflow is the same as in the [vote project](../vote/README.md), please refer to that for usage instructions. diff --git a/examples/vote-helm/api/garden.yml b/examples/vote-helm/api/garden.yml index 5f1945b44f..22bace07f0 100644 --- a/examples/vote-helm/api/garden.yml +++ b/examples/vote-helm/api/garden.yml @@ -2,13 +2,15 @@ module: description: The API backend for the voting UI type: helm name: api + base: base-chart serviceResource: - kind: Deployment containerModule: api-image dependencies: - redis values: + name: api image: + repository: api-image tag: ${modules.api-image.version} ingress: enabled: true diff --git a/examples/vote-helm/api/templates/_helpers.tpl b/examples/vote-helm/api/templates/_helpers.tpl deleted file mode 100644 index 54ca5c0215..0000000000 --- a/examples/vote-helm/api/templates/_helpers.tpl +++ /dev/null @@ -1,32 +0,0 @@ -{{/* vim: set filetype=mustache: */}} -{{/* -Expand the name of the chart. -*/}} -{{- define "api.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} -{{- end -}} - -{{/* -Create a default fully qualified app name. -We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). -If release name contains chart name it will be used as a full name. -*/}} -{{- define "api.fullname" -}} -{{- if .Values.fullnameOverride -}} -{{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- $name := default .Chart.Name .Values.nameOverride -}} -{{- if contains $name .Release.Name -}} -{{- .Release.Name | trunc 63 | trimSuffix "-" -}} -{{- else -}} -{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} -{{- end -}} -{{- end -}} -{{- end -}} - -{{/* -Create chart name and version as used by the chart label. -*/}} -{{- define "api.chart" -}} -{{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} -{{- end -}} diff --git a/examples/vote-helm/api/templates/deployment.yaml b/examples/vote-helm/api/templates/deployment.yaml deleted file mode 100644 index 93a8e6ecee..0000000000 --- a/examples/vote-helm/api/templates/deployment.yaml +++ /dev/null @@ -1,44 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {{ include "api.fullname" . }} - labels: - app.kubernetes.io/name: {{ include "api.name" . }} - helm.sh/chart: {{ include "api.chart" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/managed-by: {{ .Release.Service }} -spec: - replicas: {{ .Values.replicaCount }} - selector: - matchLabels: - app.kubernetes.io/name: {{ include "api.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - template: - metadata: - labels: - app.kubernetes.io/name: {{ include "api.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - spec: - containers: - - name: {{ .Chart.Name }} - image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" - imagePullPolicy: {{ .Values.image.pullPolicy }} - args: [python, app.py] - ports: - - name: http - containerPort: 80 - protocol: TCP - resources: - {{- toYaml .Values.resources | nindent 12 }} - {{- with .Values.nodeSelector }} - nodeSelector: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.affinity }} - affinity: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.tolerations }} - tolerations: - {{- toYaml . | nindent 8 }} - {{- end }} diff --git a/examples/vote-helm/api/.helmignore b/examples/vote-helm/base-chart/.helmignore similarity index 100% rename from examples/vote-helm/api/.helmignore rename to examples/vote-helm/base-chart/.helmignore diff --git a/examples/vote-helm/api/Chart.yaml b/examples/vote-helm/base-chart/Chart.yaml similarity index 83% rename from examples/vote-helm/api/Chart.yaml rename to examples/vote-helm/base-chart/Chart.yaml index 6b53945ade..8cbcf494f7 100644 --- a/examples/vote-helm/api/Chart.yaml +++ b/examples/vote-helm/base-chart/Chart.yaml @@ -1,5 +1,5 @@ apiVersion: v1 appVersion: "1.0" description: A Helm chart for Kubernetes -name: api +name: base-chart version: 0.1.0 diff --git a/examples/vote-helm/base-chart/garden.yml b/examples/vote-helm/base-chart/garden.yml new file mode 100644 index 0000000000..0b45f15d66 --- /dev/null +++ b/examples/vote-helm/base-chart/garden.yml @@ -0,0 +1,7 @@ +module: + description: Base Helm chart for services + type: helm + name: base-chart + serviceResource: + kind: Deployment + skipDeploy: true diff --git a/examples/vote-helm/api/templates/NOTES.txt b/examples/vote-helm/base-chart/templates/NOTES.txt similarity index 77% rename from examples/vote-helm/api/templates/NOTES.txt rename to examples/vote-helm/base-chart/templates/NOTES.txt index 9e565235ac..3af568384d 100644 --- a/examples/vote-helm/api/templates/NOTES.txt +++ b/examples/vote-helm/base-chart/templates/NOTES.txt @@ -6,16 +6,16 @@ {{- end }} {{- end }} {{- else if contains "NodePort" .Values.service.type }} - export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "api.fullname" . }}) + export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "base-chart.fullname" . }}) export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") echo http://$NODE_IP:$NODE_PORT {{- else if contains "LoadBalancer" .Values.service.type }} NOTE: It may take a few minutes for the LoadBalancer IP to be available. - You can watch the status of by running 'kubectl get svc -w {{ include "api.fullname" . }}' - export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "api.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}') + You can watch the status of by running 'kubectl get svc -w {{ include "base-chart.fullname" . }}' + export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "base-chart.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}') echo http://$SERVICE_IP:{{ .Values.service.port }} {{- else if contains "ClusterIP" .Values.service.type }} - export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "api.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") + export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "base-chart.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") echo "Visit http://127.0.0.1:8080 to use your application" kubectl port-forward $POD_NAME 8080:80 {{- end }} diff --git a/examples/vote-helm/result/templates/_helpers.tpl b/examples/vote-helm/base-chart/templates/_helpers.tpl similarity index 74% rename from examples/vote-helm/result/templates/_helpers.tpl rename to examples/vote-helm/base-chart/templates/_helpers.tpl index 9468b607e6..1acfa1f3b0 100644 --- a/examples/vote-helm/result/templates/_helpers.tpl +++ b/examples/vote-helm/base-chart/templates/_helpers.tpl @@ -2,8 +2,8 @@ {{/* Expand the name of the chart. */}} -{{- define "result.name" -}} -{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} +{{- define "base-chart.name" -}} +{{- required "Must specify name field in values" .Values.name | trunc 63 | trimSuffix "-" -}} {{- end -}} {{/* @@ -11,11 +11,11 @@ Create a default fully qualified app name. We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec). If release name contains chart name it will be used as a full name. */}} -{{- define "result.fullname" -}} +{{- define "base-chart.fullname" -}} {{- if .Values.fullnameOverride -}} {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} {{- else -}} -{{- $name := default .Chart.Name .Values.nameOverride -}} +{{- $name := required "Must specify name field in values" .Values.name -}} {{- if contains $name .Release.Name -}} {{- .Release.Name | trunc 63 | trimSuffix "-" -}} {{- else -}} @@ -27,6 +27,6 @@ If release name contains chart name it will be used as a full name. {{/* Create chart name and version as used by the chart label. */}} -{{- define "result.chart" -}} +{{- define "base-chart.chart" -}} {{- printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 | trimSuffix "-" -}} {{- end -}} diff --git a/examples/vote-helm/base-chart/templates/deployment.yaml b/examples/vote-helm/base-chart/templates/deployment.yaml new file mode 100644 index 0000000000..07472c8c31 --- /dev/null +++ b/examples/vote-helm/base-chart/templates/deployment.yaml @@ -0,0 +1,33 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: {{ include "base-chart.fullname" . }} + labels: + app.kubernetes.io/name: {{ include "base-chart.name" . }} + helm.sh/chart: {{ include "base-chart.chart" . }} + app.kubernetes.io/instance: {{ .Release.Name }} + app.kubernetes.io/managed-by: {{ .Release.Service }} +spec: + replicas: {{ .Values.replicaCount }} + selector: + matchLabels: + app.kubernetes.io/name: {{ include "base-chart.name" . }} + app.kubernetes.io/instance: {{ .Release.Name }} + template: + metadata: + labels: + app.kubernetes.io/name: {{ include "base-chart.name" . }} + app.kubernetes.io/instance: {{ .Release.Name }} + spec: + containers: + - name: {{ include "base-chart.name" . }} + image: "{{ required "Must specify image.repository field in values" .Values.image.repository }}:{{ required "Must specify image.tag field in values" .Values.image.tag }}" + imagePullPolicy: IfNotPresent + args: + {{- toYaml .Values.args | nindent 12 }} + ports: + - name: http + containerPort: 80 + protocol: TCP + resources: + {{- toYaml .Values.resources | nindent 12 }} diff --git a/examples/vote-helm/api/templates/ingress.yaml b/examples/vote-helm/base-chart/templates/ingress.yaml similarity index 84% rename from examples/vote-helm/api/templates/ingress.yaml rename to examples/vote-helm/base-chart/templates/ingress.yaml index 291fdf7349..d422c5b570 100644 --- a/examples/vote-helm/api/templates/ingress.yaml +++ b/examples/vote-helm/base-chart/templates/ingress.yaml @@ -1,13 +1,13 @@ {{- if .Values.ingress.enabled -}} -{{- $fullName := include "api.fullname" . -}} +{{- $fullName := include "base-chart.fullname" . -}} {{- $ingressPaths := .Values.ingress.paths -}} apiVersion: extensions/v1beta1 kind: Ingress metadata: name: {{ $fullName }} labels: - app.kubernetes.io/name: {{ include "api.name" . }} - helm.sh/chart: {{ include "api.chart" . }} + app.kubernetes.io/name: {{ include "base-chart.name" . }} + helm.sh/chart: {{ include "base-chart.chart" . }} app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- with .Values.ingress.annotations }} diff --git a/examples/vote-helm/api/templates/service.yaml b/examples/vote-helm/base-chart/templates/service.yaml similarity index 52% rename from examples/vote-helm/api/templates/service.yaml rename to examples/vote-helm/base-chart/templates/service.yaml index 726bdc400d..057ba99da9 100644 --- a/examples/vote-helm/api/templates/service.yaml +++ b/examples/vote-helm/base-chart/templates/service.yaml @@ -1,19 +1,18 @@ apiVersion: v1 kind: Service metadata: - name: {{ include "api.fullname" . }} + name: {{ include "base-chart.fullname" . }} labels: - app.kubernetes.io/name: {{ include "api.name" . }} - helm.sh/chart: {{ include "api.chart" . }} + app.kubernetes.io/name: {{ include "base-chart.name" . }} + helm.sh/chart: {{ include "base-chart.chart" . }} app.kubernetes.io/instance: {{ .Release.Name }} app.kubernetes.io/managed-by: {{ .Release.Service }} spec: - type: {{ .Values.service.type }} ports: - - port: {{ .Values.service.port }} + - port: {{ .Values.servicePort }} targetPort: http protocol: TCP name: http selector: - app.kubernetes.io/name: {{ include "api.name" . }} + app.kubernetes.io/name: {{ include "base-chart.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} diff --git a/examples/vote-helm/api/values.yaml b/examples/vote-helm/base-chart/values.yaml similarity index 81% rename from examples/vote-helm/api/values.yaml rename to examples/vote-helm/base-chart/values.yaml index 7bbb6c9188..d16d7a2531 100644 --- a/examples/vote-helm/api/values.yaml +++ b/examples/vote-helm/base-chart/values.yaml @@ -1,20 +1,18 @@ -# Default values for api. +# Default values for base-chart. # This is a YAML-formatted file. # Declare variables to be passed into your templates. replicaCount: 1 image: - repository: api-image - tag: stable - pullPolicy: IfNotPresent + repository: + tag: -nameOverride: "" fullnameOverride: "" -service: - type: ClusterIP - port: 80 +args: [] + +servicePort: 80 ingress: enabled: false @@ -40,9 +38,3 @@ resources: {} # requests: # cpu: 100m # memory: 128Mi - -nodeSelector: {} - -tolerations: [] - -affinity: {} diff --git a/examples/vote-helm/result/.helmignore b/examples/vote-helm/result/.helmignore deleted file mode 100644 index 50af031725..0000000000 --- a/examples/vote-helm/result/.helmignore +++ /dev/null @@ -1,22 +0,0 @@ -# Patterns to ignore when building packages. -# This supports shell glob matching, relative path matching, and -# negation (prefixed with !). Only one pattern per line. -.DS_Store -# Common VCS dirs -.git/ -.gitignore -.bzr/ -.bzrignore -.hg/ -.hgignore -.svn/ -# Common backup files -*.swp -*.bak -*.tmp -*~ -# Various IDEs -.project -.idea/ -*.tmproj -.vscode/ diff --git a/examples/vote-helm/result/Chart.yaml b/examples/vote-helm/result/Chart.yaml deleted file mode 100644 index a2fbce55f4..0000000000 --- a/examples/vote-helm/result/Chart.yaml +++ /dev/null @@ -1,5 +0,0 @@ -apiVersion: v1 -appVersion: "1.0" -description: A Helm chart for Kubernetes -name: result -version: 0.1.0 diff --git a/examples/vote-helm/result/garden.yml b/examples/vote-helm/result/garden.yml index 8e08b2f418..561a402f56 100644 --- a/examples/vote-helm/result/garden.yml +++ b/examples/vote-helm/result/garden.yml @@ -1,15 +1,17 @@ module: - description: Helm chart for the voting UI + description: Helm chart for the results UI type: helm name: result + base: base-chart serviceResource: - kind: Deployment containerModule: result-image hotReloadArgs: [nodemon, server.js] dependencies: - db-init values: + name: result image: + repository: result-image tag: ${modules.result-image.version} ingress: enabled: true @@ -19,4 +21,4 @@ module: - name: integ args: [echo, ok] dependencies: - - db-init \ No newline at end of file + - db-init diff --git a/examples/vote-helm/result/templates/NOTES.txt b/examples/vote-helm/result/templates/NOTES.txt deleted file mode 100644 index 56b00b616c..0000000000 --- a/examples/vote-helm/result/templates/NOTES.txt +++ /dev/null @@ -1,21 +0,0 @@ -1. Get the application URL by running these commands: -{{- if .Values.ingress.enabled }} -{{- range $host := .Values.ingress.hosts }} - {{- range $.Values.ingress.paths }} - http{{ if $.Values.ingress.tls }}s{{ end }}://{{ $host }}{{ . }} - {{- end }} -{{- end }} -{{- else if contains "NodePort" .Values.service.type }} - export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ include "result.fullname" . }}) - export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}") - echo http://$NODE_IP:$NODE_PORT -{{- else if contains "LoadBalancer" .Values.service.type }} - NOTE: It may take a few minutes for the LoadBalancer IP to be available. - You can watch the status of by running 'kubectl get svc -w {{ include "result.fullname" . }}' - export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ include "result.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}') - echo http://$SERVICE_IP:{{ .Values.service.port }} -{{- else if contains "ClusterIP" .Values.service.type }} - export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app.kubernetes.io/name={{ include "result.name" . }},app.kubernetes.io/instance={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}") - echo "Visit http://127.0.0.1:8080 to use your application" - kubectl port-forward $POD_NAME 8080:80 -{{- end }} diff --git a/examples/vote-helm/result/templates/deployment.yaml b/examples/vote-helm/result/templates/deployment.yaml deleted file mode 100644 index 84b8ee27be..0000000000 --- a/examples/vote-helm/result/templates/deployment.yaml +++ /dev/null @@ -1,44 +0,0 @@ -apiVersion: apps/v1 -kind: Deployment -metadata: - name: {{ include "result.fullname" . }} - labels: - app.kubernetes.io/name: {{ include "result.name" . }} - helm.sh/chart: {{ include "result.chart" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/managed-by: {{ .Release.Service }} -spec: - replicas: {{ .Values.replicaCount }} - selector: - matchLabels: - app.kubernetes.io/name: {{ include "result.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - template: - metadata: - labels: - app.kubernetes.io/name: {{ include "result.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - spec: - containers: - - name: {{ .Chart.Name }} - image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}" - imagePullPolicy: {{ .Values.image.pullPolicy }} - args: [nodemon, server.js] - ports: - - name: http - containerPort: 80 - protocol: TCP - resources: - {{- toYaml .Values.resources | nindent 12 }} - {{- with .Values.nodeSelector }} - nodeSelector: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.affinity }} - affinity: - {{- toYaml . | nindent 8 }} - {{- end }} - {{- with .Values.tolerations }} - tolerations: - {{- toYaml . | nindent 8 }} - {{- end }} diff --git a/examples/vote-helm/result/templates/ingress.yaml b/examples/vote-helm/result/templates/ingress.yaml deleted file mode 100644 index d31b35228f..0000000000 --- a/examples/vote-helm/result/templates/ingress.yaml +++ /dev/null @@ -1,40 +0,0 @@ -{{- if .Values.ingress.enabled -}} -{{- $fullName := include "result.fullname" . -}} -{{- $ingressPaths := .Values.ingress.paths -}} -apiVersion: extensions/v1beta1 -kind: Ingress -metadata: - name: {{ $fullName }} - labels: - app.kubernetes.io/name: {{ include "result.name" . }} - helm.sh/chart: {{ include "result.chart" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/managed-by: {{ .Release.Service }} - {{- with .Values.ingress.annotations }} - annotations: - {{- toYaml . | nindent 4 }} - {{- end }} -spec: -{{- if .Values.ingress.tls }} - tls: - {{- range .Values.ingress.tls }} - - hosts: - {{- range .hosts }} - - {{ . | quote }} - {{- end }} - secretName: {{ .secretName }} - {{- end }} -{{- end }} - rules: - {{- range .Values.ingress.hosts }} - - host: {{ . | quote }} - http: - paths: - {{- range $ingressPaths }} - - path: {{ . }} - backend: - serviceName: {{ $fullName }} - servicePort: http - {{- end }} - {{- end }} -{{- end }} diff --git a/examples/vote-helm/result/templates/service.yaml b/examples/vote-helm/result/templates/service.yaml deleted file mode 100644 index 032d5f0903..0000000000 --- a/examples/vote-helm/result/templates/service.yaml +++ /dev/null @@ -1,19 +0,0 @@ -apiVersion: v1 -kind: Service -metadata: - name: {{ include "result.fullname" . }} - labels: - app.kubernetes.io/name: {{ include "result.name" . }} - helm.sh/chart: {{ include "result.chart" . }} - app.kubernetes.io/instance: {{ .Release.Name }} - app.kubernetes.io/managed-by: {{ .Release.Service }} -spec: - type: {{ .Values.service.type }} - ports: - - port: {{ .Values.service.port }} - targetPort: http - protocol: TCP - name: http - selector: - app.kubernetes.io/name: {{ include "result.name" . }} - app.kubernetes.io/instance: {{ .Release.Name }} diff --git a/examples/vote-helm/result/values.yaml b/examples/vote-helm/result/values.yaml deleted file mode 100644 index 395be5f0e9..0000000000 --- a/examples/vote-helm/result/values.yaml +++ /dev/null @@ -1,48 +0,0 @@ -# Default values for result. -# This is a YAML-formatted file. -# Declare variables to be passed into your templates. - -replicaCount: 1 - -image: - repository: result-image - tag: stable - pullPolicy: IfNotPresent - -nameOverride: "" -fullnameOverride: "" - -service: - type: ClusterIP - port: 80 - -ingress: - enabled: false - annotations: {} - # kubernetes.io/ingress.class: nginx - # kubernetes.io/tls-acme: "true" - paths: [] - hosts: - - chart-example.local - tls: [] - # - secretName: chart-example-tls - # hosts: - # - chart-example.local - -resources: {} - # We usually recommend not to specify default resources and to leave this as a conscious - # choice for the user. This also increases chances charts run on environments with little - # resources, such as Minikube. If you do want to specify resources, uncomment the following - # lines, adjust them as necessary, and remove the curly braces after 'resources:'. - # limits: - # cpu: 100m - # memory: 128Mi - # requests: - # cpu: 100m - # memory: 128Mi - -nodeSelector: {} - -tolerations: [] - -affinity: {} diff --git a/garden-service/package-lock.json b/garden-service/package-lock.json index c593001013..7d92ff9175 100644 --- a/garden-service/package-lock.json +++ b/garden-service/package-lock.json @@ -651,6 +651,12 @@ "integrity": "sha512-JRDtMPEqXrzfuYAdqbxLot1GvAr/QvicIZAnOAigZaj8xVMhuSJTg/xsv9E1TvyL+wujYhRLx9ZsQ0oFOSmwyA==", "dev": true }, + "@types/json-merge-patch": { + "version": "0.0.4", + "resolved": "https://registry.npmjs.org/@types/json-merge-patch/-/json-merge-patch-0.0.4.tgz", + "integrity": "sha1-pSgtqWkKgSpiEoo0cIr0dqMI2UE=", + "dev": true + }, "@types/json-stringify-safe": { "version": "5.0.0", "resolved": "https://registry.npmjs.org/@types/json-stringify-safe/-/json-stringify-safe-5.0.0.tgz", @@ -3744,7 +3750,8 @@ "ansi-regex": { "version": "2.1.1", "resolved": false, - "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=" + "integrity": "sha1-w7M6te42DYbg5ijwRorn7yfWVN8=", + "optional": true }, "aproba": { "version": "1.2.0", @@ -3765,12 +3772,14 @@ "balanced-match": { "version": "1.0.0", "resolved": false, - "integrity": "sha1-ibTRmasr7kneFk6gK4nORi1xt2c=" + "integrity": "sha1-ibTRmasr7kneFk6gK4nORi1xt2c=", + "optional": true }, "brace-expansion": { "version": "1.1.11", "resolved": false, "integrity": "sha512-iCuPHDFgrHX7H2vEI/5xpz07zSHB00TpugqhmYtVmMO6518mCuRMoOYFldEBl0g187ufozdaHgWKcYFb61qGiA==", + "optional": true, "requires": { "balanced-match": "^1.0.0", "concat-map": "0.0.1" @@ -3785,17 +3794,20 @@ "code-point-at": { "version": "1.1.0", "resolved": false, - "integrity": "sha1-DQcLTQQ6W+ozovGkDi7bPZpMz3c=" + "integrity": "sha1-DQcLTQQ6W+ozovGkDi7bPZpMz3c=", + "optional": true }, "concat-map": { "version": "0.0.1", "resolved": false, - "integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s=" + "integrity": "sha1-2Klr13/Wjfd5OnMDajug1UBdR3s=", + "optional": true }, "console-control-strings": { "version": "1.1.0", "resolved": false, - "integrity": "sha1-PXz0Rk22RG6mRL9LOVB/mFEAjo4=" + "integrity": "sha1-PXz0Rk22RG6mRL9LOVB/mFEAjo4=", + "optional": true }, "core-util-is": { "version": "1.0.2", @@ -3912,7 +3924,8 @@ "inherits": { "version": "2.0.3", "resolved": false, - "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4=" + "integrity": "sha1-Yzwsg+PaQqUC9SRmAiSA9CCCYd4=", + "optional": true }, "ini": { "version": "1.3.5", @@ -3924,6 +3937,7 @@ "version": "1.0.0", "resolved": false, "integrity": "sha1-754xOG8DGn8NZDr4L95QxFfvAMs=", + "optional": true, "requires": { "number-is-nan": "^1.0.0" } @@ -3938,6 +3952,7 @@ "version": "3.0.4", "resolved": false, "integrity": "sha512-yJHVQEhyqPLUTgt9B83PXu6W3rx4MvvHvSUvToogpwoGDOUQ+yDrR0HRot+yOCdCO7u4hX3pWft6kWBBcqh0UA==", + "optional": true, "requires": { "brace-expansion": "^1.1.7" } @@ -3945,12 +3960,14 @@ "minimist": { "version": "0.0.8", "resolved": false, - "integrity": "sha1-hX/Kv8M5fSYluCKCYuhqp6ARsF0=" + "integrity": "sha1-hX/Kv8M5fSYluCKCYuhqp6ARsF0=", + "optional": true }, "minipass": { "version": "2.2.4", "resolved": false, "integrity": "sha512-hzXIWWet/BzWhYs2b+u7dRHlruXhwdgvlTMDKC6Cb1U7ps6Ac6yQlR39xsbjWJE377YTCtKwIXIpJ5oP+j5y8g==", + "optional": true, "requires": { "safe-buffer": "^5.1.1", "yallist": "^3.0.0" @@ -3969,6 +3986,7 @@ "version": "0.5.1", "resolved": false, "integrity": "sha1-MAV0OOrGz3+MR2fzhkjWaX11yQM=", + "optional": true, "requires": { "minimist": "0.0.8" } @@ -4049,7 +4067,8 @@ "number-is-nan": { "version": "1.0.1", "resolved": false, - "integrity": "sha1-CXtgK1NCKlIsGvuHkDGDNpQaAR0=" + "integrity": "sha1-CXtgK1NCKlIsGvuHkDGDNpQaAR0=", + "optional": true }, "object-assign": { "version": "4.1.1", @@ -4061,6 +4080,7 @@ "version": "1.4.0", "resolved": false, "integrity": "sha1-WDsap3WWHUsROsF9nFC6753Xa9E=", + "optional": true, "requires": { "wrappy": "1" } @@ -4146,7 +4166,8 @@ "safe-buffer": { "version": "5.1.1", "resolved": false, - "integrity": "sha512-kKvNJn6Mm93gAczWVJg7wH+wGYWNrDHdWvpUmHyEsgCtIwwo3bqPtV4tR5tuPaUhTOo/kvhVwd8XwwOllGYkbg==" + "integrity": "sha512-kKvNJn6Mm93gAczWVJg7wH+wGYWNrDHdWvpUmHyEsgCtIwwo3bqPtV4tR5tuPaUhTOo/kvhVwd8XwwOllGYkbg==", + "optional": true }, "safer-buffer": { "version": "2.1.2", @@ -4182,6 +4203,7 @@ "version": "1.0.2", "resolved": false, "integrity": "sha1-EYvfW4zcUaKn5w0hHgfisLmxB9M=", + "optional": true, "requires": { "code-point-at": "^1.0.0", "is-fullwidth-code-point": "^1.0.0", @@ -4201,6 +4223,7 @@ "version": "3.0.1", "resolved": false, "integrity": "sha1-ajhfuIU9lS1f8F0Oiq+UJ43GPc8=", + "optional": true, "requires": { "ansi-regex": "^2.0.0" } @@ -4244,12 +4267,14 @@ "wrappy": { "version": "1.0.2", "resolved": false, - "integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8=" + "integrity": "sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8=", + "optional": true }, "yallist": { "version": "3.0.2", "resolved": false, - "integrity": "sha1-hFK0u36Dx8GI2AQcGoN8dz1ti7k=" + "integrity": "sha1-hFK0u36Dx8GI2AQcGoN8dz1ti7k=", + "optional": true } } }, @@ -5923,6 +5948,14 @@ "integrity": "sha1-5CGiqOINawgZ3yiQj3glJrlt0f4=", "dev": true }, + "json-merge-patch": { + "version": "0.2.3", + "resolved": "https://registry.npmjs.org/json-merge-patch/-/json-merge-patch-0.2.3.tgz", + "integrity": "sha1-+ixrWvh9p3uuKWalidUuI+2B/kA=", + "requires": { + "deep-equal": "^1.0.0" + } + }, "json-parse-better-errors": { "version": "1.0.2", "resolved": "https://registry.npmjs.org/json-parse-better-errors/-/json-parse-better-errors-1.0.2.tgz", @@ -7398,6 +7431,7 @@ "resolved": "https://registry.npmjs.org/align-text/-/align-text-0.1.4.tgz", "integrity": "sha1-DNkKVhCT810KmSVsIrcGlDP60Rc=", "dev": true, + "optional": true, "requires": { "kind-of": "^3.0.2", "longest": "^1.0.1", @@ -7767,7 +7801,8 @@ "version": "1.1.6", "resolved": "https://registry.npmjs.org/is-buffer/-/is-buffer-1.1.6.tgz", "integrity": "sha512-NcdALwpXkTm5Zvvbk7owOUSvVvBKDgKP5/ewfXEznmQFfs4ZRmanOeKBTjRVjka3QFoN6XJ+9F3USqfHqTaU5w==", - "dev": true + "dev": true, + "optional": true }, "is-builtin-module": { "version": "1.0.0", @@ -7863,6 +7898,7 @@ "resolved": "https://registry.npmjs.org/kind-of/-/kind-of-3.2.2.tgz", "integrity": "sha1-MeohpzS6ubuw8yRm2JOupR5KPGQ=", "dev": true, + "optional": true, "requires": { "is-buffer": "^1.1.5" } @@ -7915,7 +7951,8 @@ "version": "1.0.1", "resolved": "https://registry.npmjs.org/longest/-/longest-1.0.1.tgz", "integrity": "sha1-MKCy2jj3N3DoKUoNIuZiXtd9AJc=", - "dev": true + "dev": true, + "optional": true }, "lru-cache": { "version": "4.1.3", @@ -8218,7 +8255,8 @@ "version": "1.6.1", "resolved": "https://registry.npmjs.org/repeat-string/-/repeat-string-1.6.1.tgz", "integrity": "sha1-jcrkcOHIirwtYA//Sndihtp15jc=", - "dev": true + "dev": true, + "optional": true }, "require-directory": { "version": "2.1.1", diff --git a/garden-service/package.json b/garden-service/package.json index 67d8636b4b..b0e084bd4e 100644 --- a/garden-service/package.json +++ b/garden-service/package.json @@ -56,6 +56,7 @@ "inquirer": "^6.2.1", "joi": "^14.3.0", "js-yaml": "^3.12.0", + "json-merge-patch": "^0.2.3", "json-stringify-safe": "^5.0.1", "klaw": "^3.0.0", "koa": "^2.6.2", @@ -106,6 +107,7 @@ "@types/inquirer": "0.0.43", "@types/joi": "^14.0.1", "@types/js-yaml": "^3.11.2", + "@types/json-merge-patch": "0.0.4", "@types/json-stringify-safe": "^5.0.0", "@types/klaw": "^2.1.1", "@types/koa": "^2.0.47", diff --git a/garden-service/src/actions.ts b/garden-service/src/actions.ts index 625479229f..5fceb84b8a 100644 --- a/garden-service/src/actions.ts +++ b/garden-service/src/actions.ts @@ -10,7 +10,7 @@ import Bluebird = require("bluebird") import chalk from "chalk" import { Garden } from "./garden" import { PrimitiveMap } from "./config/common" -import { Module, ModuleMap } from "./types/module" +import { Module } from "./types/module" import { ModuleActions, ServiceActions, PluginActions, TaskActions } from "./types/plugin/plugin" import { BuildResult, @@ -99,15 +99,15 @@ type ActionHelperParams = Omit & { pluginName?: string } type ModuleActionHelperParams = - Omit & { pluginName?: string } + Omit & { pluginName?: string } // additionally make runtimeContext param optional type ServiceActionHelperParams = - Omit + Omit & { runtimeContext?: RuntimeContext, pluginName?: string } type TaskActionHelperParams = - Omit + Omit & { runtimeContext?: RuntimeContext, pluginName?: string } type RequirePluginName = T & { pluginName: string } @@ -331,11 +331,6 @@ export class ActionHelper implements TypeGuard { //region Helper Methods //=========================================================================== - private async getBuildDependencies(module: Module): Promise { - const dependencies = await this.garden.resolveDependencyModules(module.build.dependencies, []) - return keyBy(dependencies, "name") - } - async getStatus({ log }: { log: LogEntry }): Promise { const envStatus: EnvironmentStatusMap = await this.getEnvironmentStatus({ log }) const services = keyBy(await this.garden.getServices(), "name") @@ -420,13 +415,10 @@ export class ActionHelper implements TypeGuard { defaultHandler, }) - const buildDependencies = await this.getBuildDependencies(module) - const handlerParams: any = { ...this.commonParams(handler, (params).log), ...params, module: omit(module, ["_ConfigType"]), - buildDependencies, } // TODO: figure out why this doesn't compile without the function cast return (handler)(handlerParams) @@ -447,7 +439,6 @@ export class ActionHelper implements TypeGuard { }) // TODO: figure out why this doesn't compile without the casts - const buildDependencies = await this.getBuildDependencies(module) const deps = await this.garden.getServices(service.config.dependencies) const runtimeContext = ((params).runtimeContext || await prepareRuntimeContext(this.garden, log, module, deps)) @@ -456,7 +447,6 @@ export class ActionHelper implements TypeGuard { ...params, module, runtimeContext, - buildDependencies, } return (handler)(handlerParams) @@ -480,14 +470,11 @@ export class ActionHelper implements TypeGuard { defaultHandler, }) - const buildDependencies = await this.getBuildDependencies(module) - const handlerParams: any = { ...this.commonParams(handler, (params).log), ...params, module, task, - buildDependencies, } return (handler)(handlerParams) diff --git a/garden-service/src/config/base.ts b/garden-service/src/config/base.ts index 3fbd480bc4..f9e13d2356 100644 --- a/garden-service/src/config/base.ts +++ b/garden-service/src/config/base.ts @@ -129,7 +129,6 @@ export async function loadConfig(projectRoot: string, path: string): Promise( ) { const validateOpts = { - context: `${configType} ${name ? name + " " : ""}in ${relative(projectRoot, path)}`, + context: `${configType} ${name ? name + " " : ""}(${relative(projectRoot, path)}/garden.yml)`, } if (ErrorClass) { diff --git a/garden-service/src/config/module.ts b/garden-service/src/config/module.ts index 98dd844f6b..44a8c4de08 100644 --- a/garden-service/src/config/module.ts +++ b/garden-service/src/config/module.ts @@ -12,8 +12,6 @@ import { ServiceConfig, ServiceSpec } from "./service" import { joiArray, joiIdentifier, - joiVariables, - PrimitiveMap, joiRepositoryUrl, joiUserIdentifier, } from "./common" @@ -68,7 +66,6 @@ export interface BaseModuleSpec { name: string path: string type: string - variables: PrimitiveMap repositoryUrl?: string } @@ -90,9 +87,6 @@ export const baseModuleSpecSchema = Joi.object() Garden will import the repository source code into this module, but read the module's config from the local garden.yml file.`, ), - variables: joiVariables() - .description("Variables that this module can reference and expose as environment variables.") - .example({ "my-variable": "some-value" }), allowPublish: Joi.boolean() .default(true) .description("When false, disables pushing this module to remote registries."), diff --git a/garden-service/src/garden.ts b/garden-service/src/garden.ts index b0dd2c643b..d37f46ba37 100644 --- a/garden-service/src/garden.ts +++ b/garden-service/src/garden.ts @@ -1098,12 +1098,16 @@ export class Garden { } } + /** + * This dumps the full project configuration including all modules. + */ public async dumpConfig(): Promise { const modules = await this.getModules() // Remove circular references and superfluous keys. for (const module of modules) { delete module._ConfigType + delete module.buildDependencies for (const service of module.services) { delete service.module diff --git a/garden-service/src/plugins/kubernetes/container/run.ts b/garden-service/src/plugins/kubernetes/container/run.ts index 6a7a47a879..f2966c2dc2 100644 --- a/garden-service/src/plugins/kubernetes/container/run.ts +++ b/garden-service/src/plugins/kubernetes/container/run.ts @@ -96,7 +96,7 @@ export async function runContainerModule( } export async function runContainerService( - { ctx, service, interactive, runtimeContext, timeout, log, buildDependencies }: RunServiceParams, + { ctx, service, interactive, runtimeContext, timeout, log }: RunServiceParams, ) { return runContainerModule({ ctx, @@ -106,16 +106,14 @@ export async function runContainerService( runtimeContext, timeout, log, - buildDependencies, }) } export async function runContainerTask( - { ctx, task, interactive, runtimeContext, log, buildDependencies }: RunTaskParams, + { ctx, task, interactive, runtimeContext, log }: RunTaskParams, ) { const result = await runContainerModule({ ctx, - buildDependencies, interactive, log, runtimeContext, diff --git a/garden-service/src/plugins/kubernetes/container/status.ts b/garden-service/src/plugins/kubernetes/container/status.ts index c12802eacd..95d5319e39 100644 --- a/garden-service/src/plugins/kubernetes/container/status.ts +++ b/garden-service/src/plugins/kubernetes/container/status.ts @@ -50,13 +50,12 @@ export async function waitForContainerService( runtimeContext: RuntimeContext, service: Service, hotReload: boolean, - buildDependencies, ) { const startTime = new Date().getTime() while (true) { const status = await getContainerServiceStatus({ - ctx, log, buildDependencies, service, runtimeContext, module: service.module, hotReload, + ctx, log, service, runtimeContext, module: service.module, hotReload, }) if (status.state === "ready" || status.state === "outdated") { diff --git a/garden-service/src/plugins/kubernetes/container/test.ts b/garden-service/src/plugins/kubernetes/container/test.ts index d37235bae6..b10c9560b7 100644 --- a/garden-service/src/plugins/kubernetes/container/test.ts +++ b/garden-service/src/plugins/kubernetes/container/test.ts @@ -14,7 +14,7 @@ import { runContainerModule } from "./run" import { storeTestResult } from "../test" export async function testContainerModule( - { ctx, interactive, module, runtimeContext, testConfig, log, buildDependencies }: + { ctx, interactive, module, runtimeContext, testConfig, log }: TestModuleParams, ): Promise { const testName = testConfig.name @@ -30,7 +30,6 @@ export async function testContainerModule( runtimeContext, timeout, log, - buildDependencies, }) return storeTestResult({ ctx, module, testName, result }) diff --git a/garden-service/src/plugins/kubernetes/helm/build.ts b/garden-service/src/plugins/kubernetes/helm/build.ts index 87c188d4d8..abe93b527f 100644 --- a/garden-service/src/plugins/kubernetes/helm/build.ts +++ b/garden-service/src/plugins/kubernetes/helm/build.ts @@ -9,31 +9,32 @@ import { BuildModuleParams } from "../../../types/plugin/params" import { HelmModule } from "./config" import { BuildResult } from "../../../types/plugin/outputs" -import { containsSource, getChartPath, getValuesPath } from "./common" +import { containsSource, getChartPath, getValuesPath, getBaseModule } from "./common" import { helm } from "./helm-cli" import { safeLoad } from "js-yaml" -import { set } from "lodash" import { dumpYaml } from "../../../util/util" import { join } from "path" import { GARDEN_BUILD_VERSION_FILENAME } from "../../../constants" import { writeModuleVersionFile } from "../../../vcs/base" import { LogEntry } from "../../../logger/log-entry" import { getNamespace } from "../namespace" +import { apply as jsonMerge } from "json-merge-patch" export async function buildHelmModule({ ctx, module, log }: BuildModuleParams): Promise { const buildPath = module.buildPath const namespace = await getNamespace({ ctx, provider: ctx.provider, skipCreate: true }) const context = ctx.provider.config.context + const baseModule = getBaseModule(module) - if (!(await containsSource(module))) { - log.setState("Fetching chart...") + if (!baseModule && !(await containsSource(module))) { + log.debug("Fetching chart...") try { await fetchChart(namespace, context, log, module) } catch { // update the local helm repo and retry - log.setState("Updating Helm repo...") + log.debug("Updating Helm repo...") await helm(namespace, context, log, ...["repo", "update"]) - log.setState("Fetching chart (after updating)...") + log.debug("Fetching chart (after updating)...") await fetchChart(namespace, context, log, module) } } @@ -41,14 +42,16 @@ export async function buildHelmModule({ ctx, module, log }: BuildModuleParams set(values, k, v)) + // Merge with the base module's values, if applicable + const specValues = baseModule ? jsonMerge(baseModule.spec.values, module.spec.values) : module.spec.values + + const mergedValues = jsonMerge(chartValues, specValues) const valuesPath = getValuesPath(chartPath) - await dumpYaml(valuesPath, values) + await dumpYaml(valuesPath, mergedValues) // keep track of which version has been built const buildVersionFilePath = join(buildPath, GARDEN_BUILD_VERSION_FILENAME) @@ -76,14 +79,3 @@ async function fetchChart(namespace: string, context: string, log: LogEntry, mod } await helm(namespace, context, log, ...fetchArgs) } - -// adapted from https://gist.github.com/penguinboy/762197 -function flattenValues(object, prefix = "") { - return Object.keys(object).reduce( - (prev, element) => - object[element] && typeof object[element] === "object" && !Array.isArray(object[element]) - ? { ...prev, ...flattenValues(object[element], `${prefix}${element}.`) } - : { ...prev, ...{ [`${prefix}${element}`]: object[element] } }, - {}, - ) -} diff --git a/garden-service/src/plugins/kubernetes/helm/common.ts b/garden-service/src/plugins/kubernetes/helm/common.ts index d4ef98458c..efd3759a39 100644 --- a/garden-service/src/plugins/kubernetes/helm/common.ts +++ b/garden-service/src/plugins/kubernetes/helm/common.ts @@ -6,10 +6,12 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ -import { find } from "lodash" +import { find, isEmpty } from "lodash" import { join } from "path" import { pathExists, writeFile, remove } from "fs-extra" import cryptoRandomString = require("crypto-random-string") +import { apply as jsonMerge } from "json-merge-patch" + import { PluginContext } from "../../../plugin-context" import { LogEntry } from "../../../logger/log-entry" import { getNamespace } from "../namespace" @@ -18,9 +20,10 @@ import { safeLoadAll } from "js-yaml" import { helm } from "./helm-cli" import { HelmModule, HelmModuleConfig, HelmResourceSpec } from "./config" import { HotReloadableResource } from "../hot-reload" -import { ConfigurationError } from "../../../exceptions" +import { ConfigurationError, PluginError } from "../../../exceptions" import { Module } from "../../../types/module" import { findByName } from "../../../util/util" +import { deline } from "../../../util/string" /** * Returns true if the specified Helm module contains a template (as opposed to just referencing a remote template). @@ -56,11 +59,45 @@ export async function getChartResources(ctx: PluginContext, module: Module, log: }) } +/** + * Returns the base module of the specified Helm module, or undefined if none is specified. + * Throws an error if the referenced module is missing, or is not a Helm module. + */ +export function getBaseModule(module: HelmModule) { + if (!module.spec.base) { + return + } + + const baseModule = module.buildDependencies[module.spec.base] + + if (!baseModule) { + throw new PluginError( + deline`Helm module '${module.name}' references base module '${module.spec.base}' + but it is missing from the module's build dependencies.`, + { moduleName: module.name, baseModuleName: module.spec.base }, + ) + } + + if (baseModule.type !== "helm") { + throw new ConfigurationError( + deline`Helm module '${module.name}' references base module '${module.spec.base}' + which is a '${baseModule.type}' module, but should be a helm module.`, + { moduleName: module.name, baseModuleName: module.spec.base, baseModuleType: baseModule.type }, + ) + } + + return baseModule +} + /** * Get the full path to the chart, within the module build directory. */ export async function getChartPath(module: HelmModule) { - if (await containsSource(module)) { + const baseModule = getBaseModule(module) + + if (baseModule) { + return join(module.buildPath, baseModule.spec.chartPath) + } else if (await containsSource(module)) { return join(module.buildPath, module.spec.chartPath) } else { // This value is validated to exist in the validate module action @@ -84,6 +121,32 @@ export function getReleaseName(module: HelmModule) { return module.name } +/** + * Returns the `serviceResource` spec on the module. If the module has a base module, the two resource specs + * are merged using a JSON Merge Patch (RFC 7396). + * + * Throws error if no resource spec is configured, or it is empty. + */ +export function getServiceResourceSpec(module: HelmModule) { + const baseModule = getBaseModule(module) + let resourceSpec = module.spec.serviceResource || {} + + if (baseModule) { + resourceSpec = jsonMerge(baseModule.spec.serviceResource || {}, resourceSpec) + } + + if (isEmpty(resourceSpec)) { + throw new ConfigurationError( + deline`Helm module '${module.name}' doesn't specify a \`serviceResource\` in its configuration. + You must specify a resource in the module config in order to use certain Garden features, + such as hot reloading.`, + { resourceSpec }, + ) + } + + return resourceSpec +} + interface GetServiceResourceParams { ctx: PluginContext, log: LogEntry, @@ -97,22 +160,17 @@ interface GetServiceResourceParams { * hot-reloading and other service-specific functionality. * * Optionally provide a `resourceSpec`, which is then used instead of the default `module.serviceResource` spec. + * This is used when individual tasks or tests specify a resource. * * Throws an error if no valid resource spec is given, or the resource spec doesn't match any of the given resources. */ export async function findServiceResource( { ctx, log, chartResources, module, resourceSpec }: GetServiceResourceParams, ): Promise { - if (!resourceSpec) { - resourceSpec = module.spec.serviceResource - } + const resourceMsgName = resourceSpec ? "resource" : "serviceResource" if (!resourceSpec) { - throw new ConfigurationError( - `Module '${module.name}' doesn't specify a \`serviceResource\` in its configuration. ` + - `You must specify it in the module config in order to use certain Garden features, such as hot reloading.`, - { resourceSpec }, - ) + resourceSpec = getServiceResourceSpec(module) } const targetKind = resourceSpec.kind @@ -137,23 +195,23 @@ export async function findServiceResource( if (!target) { throw new ConfigurationError( - `Module '${module.name}' does not contain specified ${targetKind} '${targetName}'`, + `Helm module '${module.name}' does not contain specified ${targetKind} '${targetName}'`, { resourceSpec, chartResourceNames }, ) } } else { if (applicableChartResources.length === 0) { throw new ConfigurationError( - `Module '${module.name}' contains no ${targetKind}s.`, + `Helm module '${module.name}' contains no ${targetKind}s.`, { resourceSpec, chartResourceNames }, ) } if (applicableChartResources.length > 1) { throw new ConfigurationError( - `Module '${module.name}' contains multiple ${targetKind}s. ` + - `You must specify \`serviceResource.name\` in the module config in order to identify ` + - `the correct ${targetKind}.`, + deline`Helm module '${module.name}' contains multiple ${targetKind}s. + You must specify \`${resourceMsgName}.name\` in the module config in order to identify + the correct ${targetKind} to use.`, { resourceSpec, chartResourceNames }, ) } diff --git a/garden-service/src/plugins/kubernetes/helm/config.ts b/garden-service/src/plugins/kubernetes/helm/config.ts index a417ee7b4c..a44d28c178 100644 --- a/garden-service/src/plugins/kubernetes/helm/config.ts +++ b/garden-service/src/plugins/kubernetes/helm/config.ts @@ -10,8 +10,16 @@ import Joi = require("joi") import { find } from "lodash" import { ServiceSpec } from "../../../config/service" -import { Primitive, joiPrimitive, joiArray, joiIdentifier, joiEnvVars, validateWithPath } from "../../../config/common" -import { Module } from "../../../types/module" +import { + Primitive, + joiPrimitive, + joiArray, + joiIdentifier, + joiEnvVars, + validateWithPath, + joiUserIdentifier, +} from "../../../config/common" +import { Module, FileCopySpec } from "../../../types/module" import { ValidateModuleParams } from "../../../types/plugin/params" import { ValidateModuleResult } from "../../../types/plugin/outputs" import { containsSource } from "./common" @@ -117,13 +125,15 @@ export const execTestSchema = baseTestSpecSchema }) export interface HelmServiceSpec extends ServiceSpec { + base?: string chart?: string chartPath: string dependencies: string[] repo?: string - serviceResource?: HelmResourceSpec, - tasks: HelmTaskSpec[], - tests: HelmTestSpec[], + serviceResource?: HelmResourceSpec + skipDeploy: boolean + tasks: HelmTaskSpec[] + tests: HelmTestSpec[] version?: string values: { [key: string]: Primitive } } @@ -137,13 +147,28 @@ const parameterValueSchema = Joi.alternatives( ) export const helmModuleSpecSchema = Joi.object().keys({ + base: joiUserIdentifier() + .description( + deline`The name of another \`helm\` module to use as a base for this one. Use this to re-use a Helm chart across + multiple services. For example, you might have an organization-wide base chart for certain types of services. + + If set, this module will by default inherit the following properties from the base module: + \`serviceResource\`, \`values\` + + Each of those can be overridden in this module. They will be merged with a JSON Merge Patch (RFC 7396).`, + ) + .example("my-base-chart"), chart: Joi.string() - .description("A valid Helm chart name or URI. Required if the module doesn't contain the Helm chart itself.") + .description( + deline`A valid Helm chart name or URI (same as you'd input to \`helm install\`). + Required if the module doesn't contain the Helm chart itself.`, + ) .example("stable/nginx-ingress"), chartPath: Joi.string() .uri({ relativeOnly: true }) .description( - "The path, relative to the module path, to the chart sources (i.e. where the Chart.yaml file is, if any).", + deline`The path, relative to the module path, to the chart sources (i.e. where the Chart.yaml file is, if any). + Not used when \`base\` is specified.`, ) .default("."), dependencies: joiArray(joiIdentifier()) @@ -160,6 +185,12 @@ export const helmModuleSpecSchema = Joi.object().keys({ We currently map a Helm chart to a single Garden service, because all the resources in a Helm chart are deployed at once.`, ), + skipDeploy: Joi.boolean() + .default(false) + .description( + deline`Set this to true if the chart should only be built, but not deployed as a service. + Use this, for example, if the chart should only be used as a base for other modules.`, + ), tasks: joiArray(execTaskSchema) .description("The task definitions for this module."), tests: joiArray(execTestSchema) @@ -184,49 +215,62 @@ export async function validateHelmModule({ ctx, moduleConfig }: ValidateModulePa projectRoot: ctx.projectRoot, }) - const { chart, chartPath, version, values, dependencies, serviceResource, tasks, tests } = moduleConfig.spec + const { + base, chart, dependencies, serviceResource, skipDeploy, tasks, tests, + } = moduleConfig.spec const sourceModuleName = serviceResource ? serviceResource.containerModule : undefined - moduleConfig.serviceConfigs = [{ - name: moduleConfig.name, - dependencies, - outputs: {}, - sourceModuleName, - spec: { chart, chartPath, version, values, dependencies, tasks, tests }, - }] + if (!skipDeploy) { + moduleConfig.serviceConfigs = [{ + name: moduleConfig.name, + dependencies, + outputs: {}, + sourceModuleName, + spec: moduleConfig.spec, + }] + } + + const containsSources = await containsSource(moduleConfig) - if (!chart && !(await containsSource(moduleConfig))) { + if (!chart && !base && !containsSources) { throw new ConfigurationError( - `Chart neither specifies a chart name, nor contains chart sources at \`chartPath\`.`, + `Chart neither specifies a chart name, base module, nor contains chart sources at \`chartPath\`.`, { moduleConfig }, ) } - // Make sure container modules specified in test+task service resources are included as build dependencies + // Make sure referenced modules are included as build dependencies // (This happens automatically for the service source module). - function checkResource(what: string, resource?: HelmResourceSpec) { - if (!resource && !serviceResource) { - throw new ConfigurationError( - deline`${what} in Helm module '${moduleConfig.name}' does not specify a target resource, - and the module does not specify a \`serviceResource\` (which would be used by default). - Please configure either of those for the configuration to be valid.`, - { moduleConfig }, - ) + function addBuildDependency(name: string, copy?: FileCopySpec[]) { + const existing = find(moduleConfig.build.dependencies, ["name", name]) + if (!copy) { + copy = [] + } + if (existing) { + existing.copy.push(...copy) + } else { + moduleConfig.build.dependencies.push({ name, copy }) } + } - if ( - resource - && resource.containerModule - && !find(moduleConfig.build.dependencies, ["name", resource.containerModule]) - ) { - moduleConfig.build.dependencies.push({ name: resource.containerModule, copy: [] }) + if (base) { + if (containsSources) { + throw new ConfigurationError(deline` + Helm module '${moduleConfig.name}' both contains sources and specifies a base module. + Since Helm charts cannot currently be merged, please either remove the sources or + the \`base\` reference in your module config. + `, { moduleConfig }) } + + // We copy the chart on build + addBuildDependency(base, [{ source: "*", target: "." }]) } moduleConfig.taskConfigs = tasks.map(spec => { - // Make sure we have a resource to run the task in - checkResource(`Task '${spec.name}'`, spec.resource) + if (spec.resource && spec.resource.containerModule) { + addBuildDependency(spec.resource.containerModule) + } return { name: spec.name, @@ -237,8 +281,9 @@ export async function validateHelmModule({ ctx, moduleConfig }: ValidateModulePa }) moduleConfig.testConfigs = tests.map(spec => { - // Make sure we have a resource to run the test suite in - checkResource(`Test suite '${spec.name}'`, spec.resource) + if (spec.resource && spec.resource.containerModule) { + addBuildDependency(spec.resource.containerModule) + } return { name: spec.name, diff --git a/garden-service/src/plugins/kubernetes/helm/deployment.ts b/garden-service/src/plugins/kubernetes/helm/deployment.ts index 54c37379c9..281093d594 100644 --- a/garden-service/src/plugins/kubernetes/helm/deployment.ts +++ b/garden-service/src/plugins/kubernetes/helm/deployment.ts @@ -12,7 +12,14 @@ import { getAppNamespace } from "../namespace" import { waitForResources } from "../status" import { helm } from "./helm-cli" import { HelmModule } from "./config" -import { getChartPath, getValuesPath, getReleaseName, getChartResources, findServiceResource } from "./common" +import { + getChartPath, + getValuesPath, + getReleaseName, + getChartResources, + findServiceResource, + getServiceResourceSpec, +} from "./common" import { getReleaseStatus, getServiceStatus } from "./status" import { configureHotReload, HotReloadableResource } from "../hot-reload" import { apply } from "../kubectl" @@ -74,7 +81,7 @@ export async function deployService( if (hotReload && hotReloadSpec && hotReloadTarget) { // Because we need to modify the Deployment, and because there is currently no reliable way to do that before // installing/upgrading via Helm, we need to separately update the target here. - const resourceSpec = module.spec.serviceResource + const resourceSpec = getServiceResourceSpec(module) configureHotReload({ target: hotReloadTarget, diff --git a/garden-service/src/plugins/kubernetes/helm/hot-reload.ts b/garden-service/src/plugins/kubernetes/helm/hot-reload.ts index 5cd7c61af9..9f92d61c93 100644 --- a/garden-service/src/plugins/kubernetes/helm/hot-reload.ts +++ b/garden-service/src/plugins/kubernetes/helm/hot-reload.ts @@ -12,7 +12,7 @@ import { deline } from "../../../util/string" import { HotReloadServiceParams } from "../../../types/plugin/params" import { ContainerModule } from "../../container/config" import { HotReloadServiceResult } from "../../../types/plugin/outputs" -import { getChartResources, findServiceResource } from "./common" +import { getChartResources, findServiceResource, getServiceResourceSpec } from "./common" import { syncToService, HotReloadableKind } from "../hot-reload" /** @@ -41,7 +41,7 @@ export async function hotReloadHelmChart( export function getHotReloadSpec(service: HelmService) { const module = service.module - const resourceSpec = module.spec.serviceResource + const resourceSpec = getServiceResourceSpec(module) if (!resourceSpec || !resourceSpec.containerModule) { throw new ConfigurationError( diff --git a/garden-service/src/plugins/kubernetes/helm/run.ts b/garden-service/src/plugins/kubernetes/helm/run.ts index 72f24bbd60..8d07595eb8 100644 --- a/garden-service/src/plugins/kubernetes/helm/run.ts +++ b/garden-service/src/plugins/kubernetes/helm/run.ts @@ -11,7 +11,7 @@ import { HelmModule, HelmResourceSpec } from "./config" import { RunTaskResult, RunResult } from "../../../types/plugin/outputs" import { getAppNamespace } from "../namespace" import { runPod } from "../run" -import { findServiceResource, getChartResources, getResourceContainer } from "./common" +import { findServiceResource, getChartResources, getResourceContainer, getServiceResourceSpec } from "./common" import { PluginContext } from "../../../plugin-context" import { LogEntry } from "../../../logger/log-entry" import { ConfigurationError } from "../../../exceptions" @@ -23,8 +23,9 @@ export async function runHelmModule( ): Promise { const context = ctx.provider.config.context const namespace = await getAppNamespace(ctx, ctx.provider) + const serviceResourceSpec = getServiceResourceSpec(module) - if (!module.spec.serviceResource) { + if (!serviceResourceSpec) { throw new ConfigurationError( `Helm module ${module.name} does not specify a \`serviceResource\`. ` + `Please configure that in order to run the module ad-hoc.`, @@ -32,7 +33,7 @@ export async function runHelmModule( ) } - const image = await getImage(ctx, module, log, module.spec.serviceResource) + const image = await getImage(ctx, module, log, serviceResourceSpec) return runPod({ context, @@ -54,7 +55,7 @@ export async function runHelmTask( const namespace = await getAppNamespace(ctx, ctx.provider) const args = task.spec.args - const image = await getImage(ctx, module, log, task.spec.resource || module.spec.serviceResource) + const image = await getImage(ctx, module, log, task.spec.resource || getServiceResourceSpec(module)) const res = await runPod({ context, diff --git a/garden-service/src/plugins/kubernetes/helm/status.ts b/garden-service/src/plugins/kubernetes/helm/status.ts index 8993b801da..f9e40aa5e5 100644 --- a/garden-service/src/plugins/kubernetes/helm/status.ts +++ b/garden-service/src/plugins/kubernetes/helm/status.ts @@ -35,12 +35,12 @@ const helmStatusCodeMap: { [code: number]: ServiceState } = { } export async function getServiceStatus( - { ctx, module, service, log, buildDependencies, hotReload }: GetServiceStatusParams, + { ctx, module, service, log, hotReload }: GetServiceStatusParams, ): Promise { // need to build to be able to check the status - const buildStatus = await getExecModuleBuildStatus({ ctx, module, log, buildDependencies }) + const buildStatus = await getExecModuleBuildStatus({ ctx, module, log }) if (!buildStatus.ready) { - await buildHelmModule({ ctx, module, log, buildDependencies }) + await buildHelmModule({ ctx, module, log }) } // first check if the installed objects on the cluster match the current code diff --git a/garden-service/src/plugins/kubernetes/helm/test.ts b/garden-service/src/plugins/kubernetes/helm/test.ts index 6a67e8f9bd..194ba89752 100644 --- a/garden-service/src/plugins/kubernetes/helm/test.ts +++ b/garden-service/src/plugins/kubernetes/helm/test.ts @@ -13,7 +13,7 @@ import { storeTestResult } from "../test" import { HelmModule } from "./config" import { getAppNamespace } from "../namespace" import { runPod } from "../run" -import { findServiceResource, getChartResources, getResourceContainer } from "./common" +import { findServiceResource, getChartResources, getResourceContainer, getServiceResourceSpec } from "./common" export async function testHelmModule( { ctx, log, interactive, module, runtimeContext, testConfig }: @@ -28,7 +28,7 @@ export async function testHelmModule( const namespace = await getAppNamespace(ctx, ctx.provider) const chartResources = await getChartResources(ctx, module, log) - const resourceSpec = testConfig.spec.resource || module.spec.serviceResource + const resourceSpec = testConfig.spec.resource || getServiceResourceSpec(module) const target = await findServiceResource({ ctx, log, chartResources, module, resourceSpec }) const container = getResourceContainer(target, resourceSpec.containerName) const image = container.image diff --git a/garden-service/src/plugins/kubernetes/hot-reload.ts b/garden-service/src/plugins/kubernetes/hot-reload.ts index ede6bad182..ee296c2633 100644 --- a/garden-service/src/plugins/kubernetes/hot-reload.ts +++ b/garden-service/src/plugins/kubernetes/hot-reload.ts @@ -158,7 +158,7 @@ export function configureHotReload({ * The hot reload action handler for containers. */ export async function hotReloadContainer( - { ctx, log, runtimeContext, service, module, buildDependencies }: HotReloadServiceParams, + { ctx, log, runtimeContext, service, module }: HotReloadServiceParams, ): Promise { const hotReloadConfig = module.spec.hotReload @@ -169,7 +169,7 @@ export async function hotReloadContainer( ) } - await waitForContainerService(ctx, log, runtimeContext, service, true, buildDependencies) + await waitForContainerService(ctx, log, runtimeContext, service, true) await syncToService(ctx, service, hotReloadConfig, "Deployment", service.name, log) return {} diff --git a/garden-service/src/plugins/local/local-docker-swarm.ts b/garden-service/src/plugins/local/local-docker-swarm.ts index d6ac378f91..bc24f45a88 100644 --- a/garden-service/src/plugins/local/local-docker-swarm.ts +++ b/garden-service/src/plugins/local/local-docker-swarm.ts @@ -40,7 +40,7 @@ export const gardenPlugin = (): GardenPlugin => ({ getServiceStatus, async deployService( - { ctx, module, service, runtimeContext, log, buildDependencies }: DeployServiceParams, + { ctx, module, service, runtimeContext, log }: DeployServiceParams, ) { // TODO: split this method up and test const { versionString } = service.module.version @@ -117,7 +117,6 @@ export const gardenPlugin = (): GardenPlugin => ({ module, runtimeContext, log, - buildDependencies, hotReload: false, }) let swarmServiceStatus @@ -175,7 +174,7 @@ export const gardenPlugin = (): GardenPlugin => ({ msg: `Ready`, }) - return getServiceStatus({ ctx, module, service, runtimeContext, log, buildDependencies, hotReload: false }) + return getServiceStatus({ ctx, module, service, runtimeContext, log, hotReload: false }) }, async getServiceOutputs({ ctx, service }: GetServiceOutputsParams) { @@ -185,7 +184,7 @@ export const gardenPlugin = (): GardenPlugin => ({ }, async execInService( - { ctx, service, command, runtimeContext, log, buildDependencies }: ExecInServiceParams, + { ctx, service, command, runtimeContext, log }: ExecInServiceParams, ) { const status = await getServiceStatus({ ctx, @@ -193,7 +192,6 @@ export const gardenPlugin = (): GardenPlugin => ({ module: service.module, runtimeContext, log, - buildDependencies, hotReload: false, }) diff --git a/garden-service/src/plugins/local/local-google-cloud-functions.ts b/garden-service/src/plugins/local/local-google-cloud-functions.ts index df18ca0856..e27ec44a7e 100644 --- a/garden-service/src/plugins/local/local-google-cloud-functions.ts +++ b/garden-service/src/plugins/local/local-google-cloud-functions.ts @@ -91,7 +91,6 @@ export const gardenPlugin = (): GardenPlugin => ({ name: parsed.name, path: parsed.path, type: "container", - variables: parsed.variables, spec: { buildArgs: { diff --git a/garden-service/src/plugins/openfaas/openfaas.ts b/garden-service/src/plugins/openfaas/openfaas.ts index 29904bd645..43a1f70a3d 100644 --- a/garden-service/src/plugins/openfaas/openfaas.ts +++ b/garden-service/src/plugins/openfaas/openfaas.ts @@ -266,7 +266,7 @@ export function gardenPlugin({ config }: { config: OpenFaasConfig }): GardenPlug }, async deleteService(params: DeleteServiceParams): Promise { - const { ctx, log, service, runtimeContext, buildDependencies } = params + const { ctx, log, service, runtimeContext } = params let status let found = true @@ -276,7 +276,6 @@ export function gardenPlugin({ config }: { config: OpenFaasConfig }): GardenPlug log, service, runtimeContext, - buildDependencies, module: service.module, hotReload: false, }) diff --git a/garden-service/src/types/module.ts b/garden-service/src/types/module.ts index 00840eac20..d4e761b74d 100644 --- a/garden-service/src/types/module.ts +++ b/garden-service/src/types/module.ts @@ -6,7 +6,7 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ -import { flatten, uniq, cloneDeep } from "lodash" +import { flatten, uniq, cloneDeep, keyBy } from "lodash" import { getNames } from "../util/util" import { TestSpec } from "../config/test" import { ModuleSpec, ModuleConfig, moduleConfigSchema } from "../config/module" @@ -18,7 +18,7 @@ import { pathToCacheContext } from "../cache" import { Garden } from "../garden" import { serviceFromConfig, Service, serviceSchema } from "./service" import * as Joi from "joi" -import { joiArray, joiIdentifier } from "../config/common" +import { joiArray, joiIdentifier, joiIdentifierMap } from "../config/common" import * as Bluebird from "bluebird" export interface FileCopySpec { @@ -35,6 +35,8 @@ export interface Module< buildPath: string version: ModuleVersion + buildDependencies: ModuleMap + services: Service>[] serviceNames: string[] serviceDependencyNames: string[] @@ -54,6 +56,9 @@ export const moduleSchema = moduleConfigSchema .description("The path to the build staging directory for the module."), version: moduleVersionSchema .required(), + buildDependencies: joiIdentifierMap(Joi.lazy(() => moduleSchema)) + .required() + .description("A map of all modules referenced under \`build.dependencies\`."), services: joiArray(Joi.lazy(() => serviceSchema)) .required() .description("A list of all the services that the module provides."), @@ -89,6 +94,8 @@ export async function moduleFromConfig(garden: Garden, config: ModuleConfig): Pr buildPath: await garden.buildDir.buildPath(config.name), version: await garden.resolveVersion(config.name, config.build.dependencies), + buildDependencies: {}, + services: [], serviceNames: getNames(config.serviceConfigs), serviceDependencyNames: uniq(flatten(config.serviceConfigs @@ -104,6 +111,11 @@ export async function moduleFromConfig(garden: Garden, config: ModuleConfig): Pr _ConfigType: config, } + const buildDependencyModules = await Bluebird.map( + module.build.dependencies, d => garden.getModule(getModuleKey(d.name, d.plugin)), + ) + module.buildDependencies = keyBy(buildDependencyModules, "name") + module.services = await Bluebird.map( config.serviceConfigs, serviceConfig => serviceFromConfig(garden, module, serviceConfig), diff --git a/garden-service/src/types/plugin/params.ts b/garden-service/src/types/plugin/params.ts index 19d88a8469..87dd45a418 100644 --- a/garden-service/src/types/plugin/params.ts +++ b/garden-service/src/types/plugin/params.ts @@ -11,7 +11,7 @@ import Stream from "ts-stream" import { LogEntry } from "../../logger/log-entry" import { PluginContext, pluginContextSchema } from "../../plugin-context" import { ModuleVersion, moduleVersionSchema } from "../../vcs/base" -import { Primitive, joiPrimitive, joiArray, joiIdentifierMap } from "../../config/common" +import { Primitive, joiPrimitive, joiArray } from "../../config/common" import { Module, moduleSchema } from "../module" import { RuntimeContext, Service, serviceSchema, runtimeContextSchema } from "../service" import { Task } from "../task" @@ -41,13 +41,10 @@ const actionParamsSchema = Joi.object() export interface PluginModuleActionParamsBase extends PluginActionParamsBase { module: T - buildDependencies: { [name: string]: Module } } const moduleActionParamsSchema = actionParamsSchema .keys({ module: moduleSchema, - buildDependencies: joiIdentifierMap(moduleSchema) - .description("All build dependencies of this module, keyed by name."), }) export interface PluginServiceActionParamsBase diff --git a/garden-service/test/helpers.ts b/garden-service/test/helpers.ts index bfb74073e8..0987646ef7 100644 --- a/garden-service/test/helpers.ts +++ b/garden-service/test/helpers.ts @@ -163,7 +163,7 @@ export const testPlugin: PluginFactory = (): GardenPlugin => { async deployService() { return {} }, async runService( - { ctx, service, interactive, runtimeContext, timeout, log, buildDependencies }: RunServiceParams, + { ctx, service, interactive, runtimeContext, timeout, log }: RunServiceParams, ) { return runModule({ ctx, @@ -173,16 +173,14 @@ export const testPlugin: PluginFactory = (): GardenPlugin => { interactive, runtimeContext, timeout, - buildDependencies, }) }, async runTask( - { ctx, task, interactive, runtimeContext, log, buildDependencies }: RunTaskParams, + { ctx, task, interactive, runtimeContext, log }: RunTaskParams, ) { const result = await runModule({ ctx, - buildDependencies, interactive, log, runtimeContext, @@ -224,7 +222,6 @@ const defaultModuleConfig: ModuleConfig = { name: "test", path: "bla", allowPublish: false, - variables: {}, build: { command: [], dependencies: [] }, spec: { services: [ diff --git a/garden-service/test/src/config/base.ts b/garden-service/test/src/config/base.ts index 043533303c..efb57f3dcd 100644 --- a/garden-service/test/src/config/base.ts +++ b/garden-service/test/src/config/base.ts @@ -73,7 +73,6 @@ describe("loadConfig", () => { allowPublish: true, build: { command: ["echo", "A"], dependencies: [] }, path: modulePathA, - variables: {}, spec: { services: [{ name: "service-a" }], diff --git a/garden-service/test/src/plugins/container.ts b/garden-service/test/src/plugins/container.ts index 8504d09cae..5ff7d5587f 100644 --- a/garden-service/test/src/plugins/container.ts +++ b/garden-service/test/src/plugins/container.ts @@ -37,7 +37,6 @@ describe("plugins.container", () => { name: "test", path: modulePath, type: "container", - variables: {}, spec: { buildArgs: {}, @@ -137,7 +136,6 @@ describe("plugins.container", () => { name: "test", path: modulePath, type: "container", - variables: {}, spec: { buildArgs: {}, @@ -191,7 +189,6 @@ describe("plugins.container", () => { name: "module-a", path: modulePath, type: "container", - variables: {}, spec: { buildArgs: {}, @@ -252,7 +249,6 @@ describe("plugins.container", () => { name: "module-a", path: modulePath, type: "container", - variables: {}, spec: { buildArgs: {}, @@ -368,7 +364,6 @@ describe("plugins.container", () => { name: "module-a", path: modulePath, type: "test", - variables: {}, spec: { buildArgs: {}, @@ -424,7 +419,6 @@ describe("plugins.container", () => { name: "module-a", path: modulePath, type: "test", - variables: {}, spec: { buildArgs: {}, @@ -475,7 +469,6 @@ describe("plugins.container", () => { name: "module-a", path: modulePath, type: "test", - variables: {}, spec: { buildArgs: {}, @@ -520,7 +513,7 @@ describe("plugins.container", () => { td.replace(containerHelpers, "imageExistsLocally", async () => true) - const result = await getBuildStatus({ ctx, log, module, buildDependencies: {} }) + const result = await getBuildStatus({ ctx, log, module }) expect(result).to.eql({ ready: true }) }) @@ -529,7 +522,7 @@ describe("plugins.container", () => { td.replace(containerHelpers, "imageExistsLocally", async () => false) - const result = await getBuildStatus({ ctx, log, module, buildDependencies: {} }) + const result = await getBuildStatus({ ctx, log, module }) expect(result).to.eql({ ready: false }) }) }) @@ -544,7 +537,7 @@ describe("plugins.container", () => { td.replace(containerHelpers, "pullImage", async () => null) td.replace(containerHelpers, "imageExistsLocally", async () => false) - const result = await build({ ctx, log, module, buildDependencies: {} }) + const result = await build({ ctx, log, module }) expect(result).to.eql({ fetched: true }) }) @@ -560,7 +553,7 @@ describe("plugins.container", () => { const dockerCli = td.replace(containerHelpers, "dockerCli") - const result = await build({ ctx, log, module, buildDependencies: {} }) + const result = await build({ ctx, log, module }) expect(result).to.eql({ fresh: true, @@ -583,7 +576,7 @@ describe("plugins.container", () => { const dockerCli = td.replace(containerHelpers, "dockerCli") - const result = await build({ ctx, log, module, buildDependencies: {} }) + const result = await build({ ctx, log, module }) expect(result).to.eql({ fresh: true, @@ -610,7 +603,7 @@ describe("plugins.container", () => { td.replace(containerHelpers, "hasDockerfile", async () => false) - const result = await publishModule({ ctx, log, module, buildDependencies: {} }) + const result = await publishModule({ ctx, log, module }) expect(result).to.eql({ published: false }) }) @@ -625,7 +618,7 @@ describe("plugins.container", () => { const dockerCli = td.replace(containerHelpers, "dockerCli") - const result = await publishModule({ ctx, log, module, buildDependencies: {} }) + const result = await publishModule({ ctx, log, module }) expect(result).to.eql({ message: "Published some/image:12345", published: true }) td.verify(dockerCli(module, ["tag", "some/image:12345", "some/image:12345"]), { times: 0 }) @@ -643,7 +636,7 @@ describe("plugins.container", () => { const dockerCli = td.replace(containerHelpers, "dockerCli") - const result = await publishModule({ ctx, log, module, buildDependencies: {} }) + const result = await publishModule({ ctx, log, module }) expect(result).to.eql({ message: "Published some/image:1.1", published: true }) td.verify(dockerCli(module, ["tag", "some/image:12345", "some/image:1.1"])) diff --git a/garden-service/test/src/plugins/kubernetes/container/ingress.ts b/garden-service/test/src/plugins/kubernetes/container/ingress.ts index 517ab39a31..546f49f317 100644 --- a/garden-service/test/src/plugins/kubernetes/container/ingress.ts +++ b/garden-service/test/src/plugins/kubernetes/container/ingress.ts @@ -339,7 +339,6 @@ describe("createIngresses", () => { name: "test", path: "/tmp", type: "container", - variables: {}, spec: { buildArgs: {}, diff --git a/garden-service/test/src/plugins/kubernetes/helm/common.ts b/garden-service/test/src/plugins/kubernetes/helm/common.ts index 40a602e773..94e750f940 100644 --- a/garden-service/test/src/plugins/kubernetes/helm/common.ts +++ b/garden-service/test/src/plugins/kubernetes/helm/common.ts @@ -11,6 +11,7 @@ import { getValuesPath, findServiceResource, getResourceContainer, + getBaseModule, } from "../../../../../src/plugins/kubernetes/helm/common" import { PluginContext } from "../../../../../src/plugin-context" import { LogEntry } from "../../../../../src/logger/log-entry" @@ -18,6 +19,7 @@ import { BuildTask } from "../../../../../src/tasks/build" import { find } from "lodash" import { deline } from "../../../../../src/util/string" import { HotReloadableResource } from "../../../../../src/plugins/kubernetes/hot-reload" +import { getServiceResourceSpec } from "../../../../../src/plugins/kubernetes/helm/common" describe("Helm common functions", () => { let garden: TestGarden @@ -430,6 +432,56 @@ describe("Helm common functions", () => { }) }) + describe("getBaseModule", () => { + it("should return undefined if no base module is specified", async () => { + const module = await garden.getModule("api") + + expect(await getBaseModule(module)).to.be.undefined + }) + + it("should return the resolved base module if specified", async () => { + const module = await garden.getModule("api") + const baseModule = await garden.getModule("postgres") + + module.spec.base = baseModule.name + module.buildDependencies = { postgres: baseModule } + + expect(await getBaseModule(module)).to.equal(baseModule) + }) + + it("should throw if the base module isn't in the build dependency map", async () => { + const module = await garden.getModule("api") + + module.spec.base = "postgres" + + await expectError( + () => getBaseModule(module), + err => expect(err.message).to.equal( + deline`Helm module 'api' references base module 'postgres' + but it is missing from the module's build dependencies.`, + ), + ) + }) + + it("should throw if the base module isn't a Helm module", async () => { + const module = await garden.getModule("api") + const baseModule = await garden.getModule("postgres") + + baseModule.type = "foo" + + module.spec.base = baseModule.name + module.buildDependencies = { postgres: baseModule } + + await expectError( + () => getBaseModule(module), + err => expect(err.message).to.equal( + deline`Helm module 'api' references base module 'postgres' which is a 'foo' module, + but should be a helm module.`, + ), + ) + }) + }) + describe("getChartPath", () => { context("module has chart sources", () => { it("should return the chart path in the build directory", async () => { @@ -463,6 +515,64 @@ describe("Helm common functions", () => { }) }) + describe("getServiceResourceSpec", () => { + it("should return the spec on the given module if it has no base module", async () => { + const module = await garden.getModule("api") + expect(await getServiceResourceSpec(module)).to.eql(module.spec.serviceResource) + }) + + it("should return the spec on the base module if there is none on the module", async () => { + const module = await garden.getModule("api") + const baseModule = await garden.getModule("postgres") + module.spec.base = "postgres" + delete module.spec.serviceResource + module.buildDependencies = { postgres: baseModule } + expect(await getServiceResourceSpec(module)).to.eql(baseModule.spec.serviceResource) + }) + + it("should merge the specs if both module and base have specs", async () => { + const module = await garden.getModule("api") + const baseModule = await garden.getModule("postgres") + module.spec.base = "postgres" + module.buildDependencies = { postgres: baseModule } + expect(await getServiceResourceSpec(module)).to.eql({ + containerModule: "api-image", + kind: "Deployment", + name: "postgres", + }) + }) + + it("should throw if there is no base module and the module has no serviceResource spec", async () => { + const module = await garden.getModule("api") + delete module.spec.serviceResource + await expectError( + () => getServiceResourceSpec(module), + err => expect(err.message).to.equal( + deline`Helm module 'api' doesn't specify a \`serviceResource\` in its configuration. + You must specify a resource in the module config in order to use certain Garden features, + such as hot reloading.`, + ), + ) + }) + + it("should throw if there is a base module but neither module has a spec", async () => { + const module = await garden.getModule("api") + const baseModule = await garden.getModule("postgres") + module.spec.base = "postgres" + module.buildDependencies = { postgres: baseModule } + delete module.spec.serviceResource + delete baseModule.spec.serviceResource + await expectError( + () => getServiceResourceSpec(module), + err => expect(err.message).to.equal( + deline`Helm module 'api' doesn't specify a \`serviceResource\` in its configuration. + You must specify a resource in the module config in order to use certain Garden features, + such as hot reloading.`, + ), + ) + }) + }) + describe("findServiceResource", () => { it("should return the resource specified by serviceResource", async () => { const module = await garden.getModule("api") @@ -478,9 +588,10 @@ describe("Helm common functions", () => { delete module.spec.serviceResource await expectError( () => findServiceResource({ ctx, log, module, chartResources }), - err => expect(err.message).to.equal(deline` - Module 'api' doesn't specify a \`serviceResource\` in its configuration. - You must specify it in the module config in order to use certain Garden features, such as hot reloading.`, + err => expect(err.message).to.equal( + deline`Helm module 'api' doesn't specify a \`serviceResource\` in its configuration. + You must specify a resource in the module config in order to use certain Garden features, + such as hot reloading.`, ), ) }) @@ -494,7 +605,7 @@ describe("Helm common functions", () => { } await expectError( () => findServiceResource({ ctx, log, module, chartResources, resourceSpec }), - err => expect(err.message).to.equal("Module 'api' contains no DaemonSets."), + err => expect(err.message).to.equal("Helm module 'api' contains no DaemonSets."), ) }) @@ -507,7 +618,7 @@ describe("Helm common functions", () => { } await expectError( () => findServiceResource({ ctx, log, module, chartResources, resourceSpec }), - err => expect(err.message).to.equal("Module 'api' does not contain specified Deployment 'foo'"), + err => expect(err.message).to.equal("Helm module 'api' does not contain specified Deployment 'foo'"), ) }) @@ -519,8 +630,9 @@ describe("Helm common functions", () => { await expectError( () => findServiceResource({ ctx, log, module, chartResources }), err => expect(err.message).to.equal(deline` - Module 'api' contains multiple Deployments. - You must specify \`serviceResource.name\` in the module config in order to identify the correct Deployment.`, + Helm module 'api' contains multiple Deployments. + You must specify \`serviceResource.name\` in the module config in order to + identify the correct Deployment to use.`, ), ) }) diff --git a/garden-service/test/src/plugins/kubernetes/helm/config.ts b/garden-service/test/src/plugins/kubernetes/helm/config.ts index 54848595bb..c14ba57341 100644 --- a/garden-service/test/src/plugins/kubernetes/helm/config.ts +++ b/garden-service/test/src/plugins/kubernetes/helm/config.ts @@ -1,10 +1,10 @@ import { resolve } from "path" import { expect } from "chai" +import { cloneDeep } from "lodash" import { TestGarden, dataDir, makeTestGarden, expectError } from "../../../../helpers" import { PluginContext } from "../../../../../src/plugin-context" import { validateHelmModule } from "../../../../../src/plugins/kubernetes/helm/config" -import { cloneDeep } from "lodash" import { deline } from "../../../../../src/util/string" describe("validateHelmModule", () => { @@ -22,8 +22,16 @@ describe("validateHelmModule", () => { await garden.close() }) + function getModuleConfig(name: string) { + const config = cloneDeep((garden).moduleConfigs[name]) + config.serviceConfigs = [] + config.taskConfigs = [] + config.testConfigs = [] + return config + } + it("should validate a Helm module", async () => { - const moduleConfig = (garden).moduleConfigs["api"] + const moduleConfig = getModuleConfig("api") const config = await validateHelmModule({ ctx, moduleConfig }) const imageModule = await garden.getModule("api-image") const { versionString } = imageModule.version @@ -50,8 +58,15 @@ describe("validateHelmModule", () => { outputs: {}, sourceModuleName: "api-image", spec: { - chart: undefined, chartPath: ".", + dependencies: [], + serviceResource: { + kind: "Deployment", + containerModule: "api-image", + }, + skipDeploy: false, + tasks: [], + tests: [], values: { image: { tag: versionString, @@ -66,18 +81,19 @@ describe("validateHelmModule", () => { ], }, }, - dependencies: [], - tasks: [], - tests: [], - version: undefined, }, }, ], spec: { + chartPath: ".", + dependencies: [], serviceResource: { kind: "Deployment", containerModule: "api-image", }, + skipDeploy: false, + tasks: [], + tests: [], values: { image: { tag: versionString, @@ -92,59 +108,88 @@ describe("validateHelmModule", () => { ], }, }, - chartPath: ".", - dependencies: [], - tasks: [], - tests: [], }, testConfigs: [], type: "helm", - variables: {}, taskConfigs: [], }) }) - it("should throw if chart contains no sources and doesn't specify chart name", async () => { - const moduleConfig = cloneDeep((garden).moduleConfigs["postgres"]) - delete moduleConfig.spec.chart - await expectError( - () => validateHelmModule({ ctx, moduleConfig }), - err => expect(err.message).to.equal(deline` - Chart neither specifies a chart name, nor contains chart sources at \`chartPath\`. - `), - ) + it("should not return a serviceConfig if skipDeploy=true", async () => { + const moduleConfig = getModuleConfig("api") + moduleConfig.spec.skipDeploy = true + const config = await validateHelmModule({ ctx, moduleConfig }) + + expect(config.serviceConfigs).to.eql([]) }) - it("should throw if a task doesn't specify resource and no serviceResource is specified", async () => { - const moduleConfig = cloneDeep((garden).moduleConfigs["api"]) - delete moduleConfig.spec.serviceResource - moduleConfig.spec.tasks = [{ - name: "foo", - args: ["foo"], - }] + it("should add the module specified under 'base' as a build dependency", async () => { + const moduleConfig = getModuleConfig("postgres") + moduleConfig.spec.base = "foo" + const config = await validateHelmModule({ ctx, moduleConfig }) + + expect(config.build.dependencies).to.eql([ + { name: "foo", copy: [{ source: "*", target: "." }] }, + ]) + }) + + it("should add copy spec to build dependency if it's already a dependency", async () => { + const moduleConfig = getModuleConfig("postgres") + moduleConfig.build.dependencies = [{ name: "foo", copy: [] }] + moduleConfig.spec.base = "foo" + const config = await validateHelmModule({ ctx, moduleConfig }) + + expect(config.build.dependencies).to.eql([ + { name: "foo", copy: [{ source: "*", target: "." }] }, + ]) + }) + + it("should add module specified under tasks[].resource.containerModule as a build dependency", async () => { + const moduleConfig = getModuleConfig("api") + moduleConfig.spec.tasks = [ + { name: "my-task", resource: { kind: "Deployment", containerModule: "foo" } }, + ] + const config = await validateHelmModule({ ctx, moduleConfig }) + + expect(config.build.dependencies).to.eql([ + { name: "api-image", copy: [] }, + { name: "foo", copy: [] }, + ]) + }) + + it("should add module specified under tests[].resource.containerModule as a build dependency", async () => { + const moduleConfig = getModuleConfig("api") + moduleConfig.spec.tests = [ + { name: "my-task", resource: { kind: "Deployment", containerModule: "foo" } }, + ] + const config = await validateHelmModule({ ctx, moduleConfig }) + + expect(config.build.dependencies).to.eql([ + { name: "api-image", copy: [] }, + { name: "foo", copy: [] }, + ]) + }) + + it("should throw if chart both contains sources and specifies base", async () => { + const moduleConfig = getModuleConfig("api") + moduleConfig.spec.base = "foo" await expectError( () => validateHelmModule({ ctx, moduleConfig }), err => expect(err.message).to.equal(deline` - Task 'foo' in Helm module 'api' does not specify a target resource, and the module does not specify a - \`serviceResource\` (which would be used by default). - Please configure either of those for the configuration to be valid. + Helm module 'api' both contains sources and specifies a base module. + Since Helm charts cannot currently be merged, please either remove the sources or + the \`base\` reference in your module config. `), ) }) - it("should throw if a test doesn't specify resource and no serviceResource is specified", async () => { - const moduleConfig = cloneDeep((garden).moduleConfigs["api"]) - delete moduleConfig.spec.serviceResource - moduleConfig.spec.tests = [{ - name: "foo", - args: ["foo"], - }] + it("should throw if chart contains no sources and doesn't specify chart name nor base", async () => { + const moduleConfig = getModuleConfig("postgres") + delete moduleConfig.spec.chart await expectError( () => validateHelmModule({ ctx, moduleConfig }), err => expect(err.message).to.equal(deline` - Test suite 'foo' in Helm module 'api' does not specify a target resource, and the module does not specify a - \`serviceResource\` (which would be used by default). - Please configure either of those for the configuration to be valid. + Chart neither specifies a chart name, base module, nor contains chart sources at \`chartPath\`. `), ) }) diff --git a/garden-service/test/src/plugins/kubernetes/helm/status.ts b/garden-service/test/src/plugins/kubernetes/helm/status.ts index 68b7919999..4f6f596c8c 100644 --- a/garden-service/test/src/plugins/kubernetes/helm/status.ts +++ b/garden-service/test/src/plugins/kubernetes/helm/status.ts @@ -13,7 +13,7 @@ describe("getServiceOutputs", () => { const service = await garden.getService("api") const module = service.module - const result = await getServiceOutputs({ ctx, module, service, buildDependencies: {}, log: garden.log }) + const result = await getServiceOutputs({ ctx, module, service, log: garden.log }) expect(result).to.eql({ "release-name": "api" }) })