feat(k8s): support remote container registries and remote clusters

This changes the deployment mechanics to include a push step before
deploying, which is performed if the provider has a `pushModule`
handler. In turn the existing "push" mechanics have been renamed to
"publish", which more accurately describes the previous implementation.

docs/guides/remote-kubernetes.md

@@ -1,32 +1,44 @@
 # Running against a remote Kubernetes cluster
 
+Below are some notes on steps you need to take before deploying Garden projects to a remote Kubernetes cluster.
+
+Many of the steps are not specific to Garden as such, so you may have already performed some of these steps
+and/or may need to follow the provided links in each section for details on how to perform the steps you have
+not yet completed.
+
 ## Setup
 
 ### Connecting to the cluster
 
 Start by making sure you have a [kubectl context](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/)
-set up on your development machine to access your cluster.
+set up on your development machine to access your cluster. How you set this up will vary by how and where you
+have deployed your cluster.
 
-Then configure the project and provider, along with the kubectl context you use to connect to your
-cluster.
+Then configure the project and provider in your project `garden.yml`, along with the kubectl context you use to
+connect to your cluster.
 
 Example:
 
 ```yaml
-    project:
-      name: my-project
-      environments:
-      - name: dev
-        providers:
-        - name: kubernetes
-          context: my-dev-context   # the name of the kubectl context for the cluster
-      defaultEnvironment: dev
+project:
+  name: my-project
+  environments:
+  - name: dev
+    providers:
+    - name: kubernetes
+      context: my-dev-context   # the name of the kubectl context for the cluster
+      ...
+  defaultEnvironment: dev
 ```
 
 ### Ingress, TLS and DNS
 
 The cluster needs to have a configured [nginx ingress controller](https://github.com/kubernetes/ingress-nginx).
-You'll also need to configure one or more TLS certificates for the hostnames you will expose for ingress.
+
+You'll also need to point one or more DNS entries to your cluster, and configure a TLS certificate for the hostnames
+you will expose for ingress.
+_How you configure DNS and prepare the certificates will depend on how you manage DNS and certificates in general,
+so we won't cover that in detail here._
 
 Once you have the certificates on hand (the `.crt` and `.key` files), create a
 [Secret](https://kubernetes.io/docs/concepts/configuration/secret/) for each cert in the cluster so that
@@ -36,37 +48,83 @@ they can be referenced when deploying services:
 kubectl create secret tls mydomain-tls-secret --key <path-to-key-file> --cert <path-to-crt-file>
 ```
 
-Then configure each certificate/secret in your provider configuration:
+Then configure each certificate/secret in your `garden.yml` provider configuration:
+
+```yaml
+project:
+  name: my-project
+  environments:
+  - name: dev
+    providers:
+    - name: kubernetes
+      context: my-dev-context
+      tlsCertificates:
+      - name: main
+        # Optionally set particular hostnames to use this certificate for
+        # (useful if you have multiple certs for the same hostname).
+        hostnames: [mydomain.com]
+        secretRef:
+          # Change to whatever name you chose for the secret above.
+          name: my-tls-secret
+          # Change this if you store the secret in another namespace.
+          namespace: default
+      - name: wildcard
+        secretRef:
+          name: wildcard-tls-secret
+          namespace: default
+      ...
+  defaultEnvironment: dev
+```
+
+### Configuring a container registry
+
+When you deploy to the environment (via `garden deploy` or `garden dev`), containers are first built and then pushed
+to the configured _deployment registry_, where the K8s cluster will then pull the built images when deploying.
+This should generally be a _private_ container registry, or at least a private project in a public registry.
+
+Similarly to the above TLS configuration, you may also need to set up auth for the registry using K8s Secrets, in this
+case via the `kubectl create secret docker-registry` helper.
+
+_Note that you do not need to configure the authentication and imagePullSecrets when using GKE along with GCR,
+as long as your deployment registry is in the same project as the GKE cluster._
+
+The lovely folks at [Heptio](https://heptio.com) have prepared good guides on how to configure private registries
+for Kubernetes, which you can find [here](http://docs.heptio.com/content/private-registries.html).
+
+Once you've created the auth secret in the cluster, you can configure the registry and the secrets in your
+`garden.yml` project config like this:
 
 ```yaml
-    project:
-      name: my-project
-      environments:
-      - name: dev
-        providers:
-        - name: kubernetes
-          context: my-dev-context
-          tlsCertificates:
-          - name: main
-            # Optionally set particular hostnames to use this certificate for
-            # (useful if you have multiple certs for the same hostname).
-            hostnames: [mydomain.com]
-            secretRef:
-              # Change to whatever name you chose for the secret above.
-              name: my-tls-secret
-              # Change this if you store the secret in another namespace.
-              namespace: default
-          - name: wildcard
-            secretRef:
-              name: wildcard-tls-secret
-              namespace: default
-      defaultEnvironment: dev
+project:
+  name: my-project
+  environments:
+  - name: dev
+    providers:
+    - name: kubernetes
+      context: my-dev-context
+      ...
+      deploymentRegistry:
+        # The hostname of the registry, e.g. gcr.io for GCR (Google Container Registry)
+        hostname: my.registry.io
+        # Namespace (aka project ID) to use in the registry for this project.
+        # For GKE/GCR, use the project ID where your cluster is.
+        namespace: my-project-id
+      imagePullSecrets:
+        # The name of the secret you stored using `kubectl create secret docker-registry`
+      - name: my-registry-secret
+        # Change this if you store the secret in another namespace.
+        namespace: default
+  defaultEnvironment: dev
 ```
 
+You also need to login to the `docker` CLI, so that images can be pushed to the registry. Please refer
+to your registry's documentation on how to do that (for Docker Hub you simply run `docker login`).
+
 ### Permissions
 
 Note that you need to have permissions to create namespaces and to create deployments,
-daemonsets, services and ingresses within the namespaces created. The plugin will
-create two or more namespaces per user and project, one to run services, another to manage
+daemonsets, services and ingresses within the namespaces created.
+
+The plugin will create two or more namespaces per user and project, one to run services, another to manage
 metadata and configuration (this is so that your environment can be reset without
 clearing your configuration variables), and potentially more to support specific plugins/providers.

docs/reference/commands.md

@@ -430,36 +430,36 @@ Examples:
 | -------- | ----- | ---- | ----------- |
   | `--tail` | `-t` | boolean | Continuously stream new logs from the service(s).
 
-### garden push
+### garden publish
 
-Build and push built module(s) to remote registry.
+Build and publish module(s) to a remote registry.
 
-Pushes built module artifacts for all or specified modules.
+Publishes built module artifacts for all or specified modules.
 Also builds modules and dependencies if needed.
 
 Examples:
 
-    garden push                # push artifacts for all modules in the project
-    garden push my-container   # only push my-container
-    garden push --force-build  # force re-build of modules before pushing artifacts
-    garden push --allow-dirty  # allow pushing dirty builds (which usually triggers error)
+    garden publish                # publish artifacts for all modules in the project
+    garden publish my-container   # only publish my-container
+    garden publish --force-build  # force re-build of modules before publishing artifacts
+    garden publish --allow-dirty  # allow publishing dirty builds (which by default triggers error)
 
 ##### Usage
 
-    garden push [module] [options]
+    garden publish [module] [options]
 
 ##### Arguments
 
 | Argument | Required | Description |
 | -------- | -------- | ----------- |
-  | `module` | No | The name of the module(s) to push (skip to push all modules). Use comma as separator to specify multiple modules.
+  | `module` | No | The name of the module(s) to publish (skip to publish all modules). Use comma as separator to specify multiple modules.
 
 ##### Options
 
 | Argument | Alias | Type | Description |
 | -------- | ----- | ---- | ----------- |
-  | `--force-build` |  | boolean | Force rebuild of module(s) before pushing.
-  | `--allow-dirty` |  | boolean | Allow pushing dirty builds (with untracked/uncommitted files).
+  | `--force-build` |  | boolean | Force rebuild of module(s) before publishing.
+  | `--allow-dirty` |  | boolean | Allow publishing dirty builds (with untracked/uncommitted changes).
 
 ### garden run module
 

docs/reference/config.md

@@ -56,7 +56,7 @@ module:
   # Set to false to disable pushing this module to remote registries.
   #
   # Optional.
-  allowPush: true
+  allowPublish: true
 
   # Specify how to build the module. Note that plugins may specify additional keys on this object.
   #
@@ -246,7 +246,7 @@ module:
   # Set to false to disable pushing this module to remote registries.
   #
   # Optional.
-  allowPush: true
+  allowPublish: true
 
   # Specify how to build the module. Note that plugins may specify additional keys on this object.
   #
@@ -380,7 +380,7 @@ module:
   # Set to false to disable pushing this module to remote registries.
   #
   # Optional.
-  allowPush: true
+  allowPublish: true
 
   # Specify how to build the module. Note that plugins may specify additional keys on this object.
   #

examples/local-tls/README.md

@@ -42,7 +42,8 @@ file, but you may also edit it directly (it's at `/etc/hosts` on most platforms)
 
 ## Usage
 
-Once you've completed the above, you can run deploy example project and the exposed ingress endpoints will be secured with TLS!
+Once you've completed the above, you can deploy the example project and the exposed ingress endpoints will be
+secured with TLS!
 
 Deploy the project:
 

examples/remote-k8s/README.md

@@ -0,0 +1,70 @@
+# Remote Kubernetes example project
+
+This project shows how you can configure Garden to work against a remote Kubernetes cluster, in addition to a local
+cluster.
+
+The example follows the [Remote Kubernetes guide](https://docs.garden.io/guides/remote-kubernetes.md). Please look
+at the guide for more details on how to configure your own project.
+
+## Setup
+
+### Prerequisites
+
+You need to have a running Kubernetes cluster, that you have API access to, and that has an exposed _nginx_ ingress
+controller. If you haven't already, you'll need to configure a `kubectl` context that has access to your cluster.
+Please refer to your cluster provider for how to do that.
+
+If you don't already have one, you also need to configure a private container registry. You could for example use
+[quay.io](https://quay.io), create a private registry on [Docker Hub](https://hub.docker.io), or use the registry
+provided by your cloud provider.
+
+### Step 1 - Update the context and cluster hostname in your config
+
+You need to update the `remote` environment configuration in your project `garden.yml`.
+Replace `my-context` with your configured `kubectl` context for the remote cluster, and `mycluster.example.com`
+with a hostname that points to the ingress controller on your cluster.
+
+### Step 2 - Get a certificate for your cluster hostname
+
+How you do this will depend on how you generally manage DNS. Basically, you need a valid TLS certificate and key for
+the hostname you configured above. If you don't have a prior preference on how to create certificates, we suggest using
+Let's Encrypt's [certbot](https://certbot.eff.org) and using the `certonly` option to generate certs.
+
+### Step 3 - Configure the certificate in your Kubernetes installation
+
+Create a Kubernetes Secret with your generated certificate and key (replace the filenames appropriately).
+
+```sh
+kubectl create secret tls garden-example --key my.key --cert my.crt
+```
+
+### Step 4 - Configure a remote container registry
+
+Each different container registry will provide different instructions on how to configure Kubernetes to authenticate.
+You can also use [Heptio's guides](http://docs.heptio.com/content/private-registries.html) for configuring private
+registries.
+
+You'll also need to login to the `docker` CLI, so that images can be pushed to the registry. Please refer
+to your registry's documentation on how to do that (for Docker Hub you simply run `docker login`).
+
+_Note that if you're using GKE along with GCR, and your deployment registry is in the same project as the GKE cluster,
+you can remove the `imagePullSecrets` section from the `garden.yml` and skip creating the auth secret._
+
+When storing the registry authorization via `kubectl create secret docker-registry`, use the name
+`garden-example-registry-auth` for the secret, and place it in the `default` namespace (or update the `garden.yml`
+configuration with the name and namespace you chose).
+
+## Usage
+
+Once you've completed the above, you can run deploy the project to the `remote` environment, by setting the
+`--env` flag when running `garden` (or you can change the `defaultEnvironment` entry in your `garden.yml`):
+
+```sh
+garden --env remote deploy
+```
+
+And then try sending a simple request using:
+
+```sh
+garden --env remote call node-service/hello
+```

examples/remote-k8s/garden.yml

@@ -0,0 +1,25 @@
+project:
+  name: local-tls
+  defaultEnvironment: local
+  environments:
+  - name: local
+    providers:
+    - name: local-kubernetes
+  # you can run garden against this environment by adding "--env remote" to your commands,
+  # e.g. garden --env remote deploy
+  - name: remote
+    providers:
+    - name: kubernetes
+      context: my-remote-context
+      defaultHostname: my-cluster.example.com   # change this to the hostname that points to your cluster
+      forceSsl: true
+      tlsCertificates:
+      - name: garden-example-tls
+        secretRef:
+          name: garden-example-tls
+          namespace: default
+      deploymentRegistry:
+        # The hostname of the registry, e.g. gcr.io for GCR (Google Container Registry)
+        hostname: my.registry.io
+        # Namespace to use in the registry for this project. For GCR, use the project ID where your cluster is.
+        namespace: my-project-id

examples/remote-k8s/services/go-service/.dockerignore

@@ -0,0 +1,4 @@
+node_modules
+Dockerfile
+garden.yml
+app.yaml

examples/remote-k8s/services/go-service/.gitignore

@@ -0,0 +1,27 @@
+# Compiled Object files, Static and Dynamic libs (Shared Objects)
+*.o
+*.a
+*.so
+
+# Folders
+_obj
+_test
+
+# Architecture specific extensions/prefixes
+*.[568vq]
+[568vq].out
+
+*.cgo1.go
+*.cgo2.c
+_cgo_defun.c
+_cgo_gotypes.go
+_cgo_export.*
+
+_testmain.go
+
+*.exe
+*.test
+*.prof
+
+.vscode/settings.json
+webserver/*server*

examples/remote-k8s/services/go-service/Dockerfile

@@ -0,0 +1,14 @@
+FROM golang:1.8.3-alpine
+MAINTAINER Aurelien PERRIER <[email protected]>
+
+ENV webserver_path /go/src/github.com/perriea/webserver/
+ENV PATH $PATH:$webserver_path
+
+WORKDIR $webserver_path
+COPY webserver/ .
+
+RUN go build .
+
+ENTRYPOINT ./webserver
+
+EXPOSE 8080