feat(k8s): add mechanism for cleaning up unused images in clusters

This required some mechanism for plugins to expose commands. We should
later iterate on that, but it's a start.

The cleanup flow and command itself is described in the
In-Cluster Building guide.

docs/README.md

@@ -8,6 +8,7 @@
   * [Features and usage](./using-garden/features-and-usage.md)
   * [Configuration files](./using-garden/configuration-files.md)
   * [Remote Clusters](./using-garden/remote-clusters.md)
+  * [In-cluster building](./using-garden/in-cluster-building.md)
   * [Using Helm charts](./using-garden/using-helm-charts.md)
   * [Hot Reload](./using-garden/hot-reload.md)
 * [Example projects](./examples/README.md)

docs/reference/commands.md

@@ -454,6 +454,36 @@ Prints all global options (options that can be applied to any command).
 
     garden options 
 
+### garden plugins
+
+Plugin-specific commands.
+
+Execute a command defined by a plugin in your project.
+Run without arguments to get a list of all plugin commands available.
+Run with just the plugin name to get a list of commands provided by that plugin.
+
+Examples:
+
+    # Run the `cleanup-cluster-registry` command from the `kubernetes` plugin.
+    garden plugins kubernetes cleanup-cluster-registry
+
+    # List all available commands.
+    garden plugins
+
+    # List all the commands from the `kubernetes` plugin.
+    garden plugins kubernetes
+
+##### Usage
+
+    garden plugins [plugin] [command] 
+
+##### Arguments
+
+| Argument | Required | Description |
+| -------- | -------- | ----------- |
+  | `plugin` | No | The name of the plugin, whose command you wish to run.
+  | `command` | No | The name of the command to run.
+
 ### garden publish
 
 Build and publish module(s) to a remote registry.

docs/using-garden/README.md

@@ -10,7 +10,14 @@ This one is all about Garden's configuration files—an overview of project and
 
 ## [Remote Clusters](./remote-clusters.md)
 
-Most of these guides currently focus on local development. If you'd like to use a remote cluster, though, check out this guide.
+Garden can work smoothly with both local and remote clusters. If you'd like to use a remote cluster, you may have some
+additional considerations and requirements. Take a look at this guide for details.
+
+## [In-cluster building](./in-cluster-building.md)
+
+One of Garden's most powerful features is the ability to build images in your Kubernetes development cluster, thus
+avoiding the need for local Kubernetes clusters. This guide covers the requirements for in-cluster building and how
+to set it up.
 
 ## [Hot Reload](./hot-reload.md)
 

docs/using-garden/in-cluster-building.md

@@ -0,0 +1,143 @@
+# Building images in remote clusters
+
+One of Garden's most powerful features is the ability to build images in your Kubernetes development cluster, thus
+avoiding the need for local Kubernetes clusters. This guide covers the requirements for in-cluster building and how
+to set it up.
+
+This guide assumes you've already read through the [Remote Clusters](./remote-clusters.md) guide.
+
+## Security considerations
+
+First off, you should only use in-cluster building in development clusters! Production clusters should not run the
+builder services for multiple reasons, both to do with resource and security concerns.
+
+You should also avoid using in-cluster building in clusters where you don't control/trust all the code being deployed,
+i.e. multi-tenant setups (where tenants are external, or otherwise not fully trusted).
+
+## Requirements
+
+In-cluster building works with _most_ Kubernetes clusters, provided they have enough resources allocated. We have
+tested on GKE, AKS, EKS and some custom installations. One provider that is currently known _not to work_ is
+DigitalOcean (track [issue #877](https://github.com/garden-io/garden/issues/877) for details and progress).
+
+Specifically, the clusters need the following:
+
+- Support for `hostPort`, and for reaching `hostPort`s from the node/Kubelet. This should work out-of-the-box in most
+  standard setups, but clusters using Cilium for networking may need to configure this specifically, for example.
+- At least 2GB of RAM _on top of your own service requirements_. More RAM is strongly recommended if you have many
+  concurrent developers or CI builds.
+- Support for `PersistentVolumeClaim`s and enough disk space for layer caches and the in-cluster image registry.
+
+You can—_and should_—adjust the allocated resources and storage in the provider configuration, under
+[resources](../reference/providers/kubernetes.md#providers[].resources) and
+[storage](../reference/providers/kubernetes.md#providers[].storage). See the individual modes below as well for more
+information on how to allocate resources appropriately.
+
+## Build modes
+
+Garden supports multiple methods for building images and making them available to the cluster:
+
+1. Cluster Docker
+2. Kaniko
+3. Local Docker
+
+The _Cluster Docker_ and _Kaniko_ modes build container images inside your development cluster, so you don't need to
+run Docker on your machine, and avoid having to build locally and push build artifacts over the wire to the cluster
+for every change to your code.
+
+The _Local Docker_ mode is the default. You should definitely use that when using _Docker for Desktop_, _Minikube_
+and most other local development clusters, and also if you're using Garden to deploy to staging/production clusters
+(more on [security considerations](#security-considerations) above).
+
+Let's look at how each mode works, and how you configure them:
+
+### Cluster Docker
+
+The Cluster Docker mode installs a standalone Docker daemon into your cluster, that is then used for builds across
+all users of the clusters, along with a handful of other supporting services. Enable this mode by setting
+`buildMode: cluster-docker` in your `kubernetes` provider configuration.
+
+In this mode, builds are executed as follows:
+
+1. Your code (build context) is synchronized to a sync service in the cluster, making it available to the
+   Docker daemon.
+2. A build is triggered in the Docker daemon.
+3. The built image is pushed to an in-cluster registry (which is automatically installed), which makes it available
+   to the cluster.
+
+After enabling this mode (we currently still default to the `local` mode), you will need to run `garden init` for each
+applicable environment, in order to install the
+required cluster-wide services. Those services include the Docker daemon itself, as well as an image registry,
+a sync service for receiving build contexts, two persistent volumes, an NFS volume provisioner for one of those volumes,
+and a couple of small utility services.
+
+Make sure your cluster has enough resources and storage to support the required services, and keep in mind that these
+services are shared across all users of the cluster. Please look at the
+[resources](../reference/providers/kubernetes.md#providers[].resources) and
+[storage](../reference/providers/kubernetes.md#providers[].storage) sections in the provider reference for
+details.
+
+### Kaniko
+
+This mode works _mostly_ the same way as Cluster Docker, but replaces the Docker daemon with
+[Kaniko](https://github.com/GoogleContainerTools/kaniko).
+Enable this by setting `buildMode: kaniko` in your `kubernetes` provider configuration.
+
+The Kaniko project is still improving, but it provides a
+compelling alternative to the standard Docker daemon because it can run without special privileges on the cluster,
+and is thus more secure. It may also scale better because it doesn't rely on a single daemon shared across users, so
+builds are executed in individual Pods and don't share the same resources of a single Pod. This also removes the need
+to provision another persistent volume, which the Docker daemon needs for its layer cache.
+
+The trade-off is generally in performance, at least for the moment, partly because it relies on the Docker registry to
+cache layers. There are also some known issues and incompatibilities, so your mileage may vary.
+
+Note the difference in how resources for the builder are allocated. See the
+[builder resources](../reference/providers/kubernetes.md#providers[].resources.builder) reference for details.
+
+### Local Docker
+
+This is the default mode. It is the least efficient one for remote clusters, but requires no additional services to be
+deployed to the cluster. For remote clusters, you do however need to explicitly configure a _deployment registry_, and
+to have Docker running locally.
+
+See the [Local Docker builds](./remote-clusters.md) section in the Remote Clusters guide for details.
+
+## Publishing images
+
+You can publish images that have been built in your cluster, using the `garden publish` command.
+
+The only caveat is that you currently need to have Docker running locally, and you need to have authenticated with the
+target registry. When publishing, we pull the image from the in-cluster registry to the local Docker daemon, and then
+go on to push it from there. We do this to avoid having to (re-)implement all the various authentication methods (and
+by extension key management) involved in pushing directly from the cluster.
+
+As usual, you need to specify the `image` field on the `container` module in question. For example:
+
+```yaml
+kind: Module
+name: my-module
+image: my-repo/my-image:v1.2.3   # <- omit the tag here if you'd like to use the Garden-generated version tag
+...
+```
+
+## Cleaning up cached images
+
+In order to avoid disk-space issues in the cluster, the `kubernetes` provider exposes a utility command:
+
+```sh
+garden --env=<your-environment> plugins kubernetes cluster-registry-cleanup
+```
+
+The command does the following:
+
+1. Looks through all Pods in the cluster to see which images/tags are in use, and flags all other images as deleted in
+   the in-cluster registry.
+2. Restarts the registry in read-only mode.
+3. Runs the registry garbage collection.
+4. Restarts the registry again without the read-only mode.
+5. When using the `cluster-docker` build mode, we additionally untag in the Docker daemon all images that are no longer
+   in the registry, and then clean up the dangling image layers by running `docker image prune`.
+
+There are plans to do this automatically when disk-space runs low, but for now you can run this manually or set up
+your own cron jobs.

docs/using-garden/remote-clusters.md

@@ -41,71 +41,21 @@ clearing your configuration variables), and potentially more to support specific
 
 ## Building and pushing images
 
-Garden supports multiple methods for building images and making them available to the cluster:
+Garden supports multiple methods for building images and making them available to the cluster. Below we detail how
+to configure for the standard out-of-cluster build flow, but do make sure to look at the
+[in-cluster building](./in-cluster-building.md) for details on how to build images directly inside the cluster.
 
-1. Cluster Docker
-2. Kaniko
-3. Local Docker
+### Local Docker builds
 
-The _Cluster Docker_ and _Kaniko_ modes build container images inside your development cluster, so you don't need to
-run Docker on your machine, and avoid having to build locally and push build artifacts over the wire to the cluster
-for every change to your code.
+This is the default build mode. It is the least efficient one for remote clusters, but requires no additional services
+to be deployed to the cluster. For remote clusters, you do however need to explicitly configure a _deployment registry_,
+and to have Docker running locally. For development clusters, you may in fact get set up quicker if you use
+[in-cluster building](./in-cluster-building.md).
 
-Let's look at how each mode works, and how you configure them:
-
-### Cluster Docker
-
-The Cluster Docker mode installs a standalone Docker daemon into your cluster, that is then used for builds across
-all users of the clusters, along with a handful of other supporting services. Enable this mode by setting
-`buildMode: cluster-docker` in your `kubernetes` provider configuration.
-
-In this mode, builds are executed as follows:
-
-1. Your code (build context) is synchronized to a sync service in the cluster, making it available to the
-   Docker daemon.
-2. A build is triggered in the Docker daemon.
-3. The built image is pushed to an in-cluster registry (which is automatically installed), which makes it available
-   to the cluster.
-
-After enabling this mode (we currently still default to the `local` mode), you will need to run `garden init` for each
-applicable environment, in order to install the
-required cluster-wide services. Those services include the Docker daemon itself, as well as an image registry,
-a sync service for receiving build contexts, two persistent volumes, an NFS volume provisioner for one of those volumes,
-and a couple of small utility services.
-
-Make sure your cluster has enough resources and storage to support the required services, and keep in mind that these
-services are shared across all users of the cluster. Please look at the
-[resources](../reference/providers/kubernetes.md#providers[].resources) and
-[storage](../reference/providers/kubernetes.md#providers[].storage) sections in the provider reference for
-details.
-
-### Kaniko
-
-This mode works _mostly_ the same way as Cluster Docker, but replaces the Docker daemon with
-[Kaniko](https://github.com/GoogleContainerTools/kaniko).
-Enable this by setting `buildMode: kaniko` in your `kubernetes` provider configuration.
-
-The Kaniko project is still improving, but it provides a
-compelling alternative to the standard Docker daemon because it can run without special privileges on the cluster,
-and is thus more secure. It may also scale better because it doesn't rely on a single daemon shared across users, so
-builds are executed in individual Pods and don't share the same resources of a single Pod. This also removes the need
-to provision another persistent volume, which the Docker daemon needs for its layer cache.
-
-The trade-off is generally in performance, at least for the moment, partly because it relies on the Docker registry to
-cache layers. There are also some known issues and incompatibilities, so your mileage may vary.
-
-Note the difference in how resources for the builder are allocated. See the
-[builder resources](../reference/providers/kubernetes.md#providers[].resources.builder) reference for details.
-
-### Local Docker
-
-This is the default mode. It is the least efficient one for remote clusters, but requires no additional services to be
-deployed to the cluster. For remote clusters, you do however need to explicitly configure a _deployment registry_, and
-to have Docker running locally.
-
-When you deploy to the environment (via `garden deploy` or `garden dev`), images are first built locally and then
-pushed to the configured deployment registry, where the K8s cluster will then pull the built images when deploying.
-This should generally be a _private_ container registry, or at least a private project in a public registry.
+When you deploy to your environment (via `garden deploy` or `garden dev`) using the local Docker mode, images are first
+built locally and then pushed to the configured _deployment registry_, where the K8s cluster will then pull the built
+images when deploying. This should generally be a _private_ container registry, or at least a private project in a
+public registry.
 
 Similarly to the below TLS configuration, you may also need to set up auth for the registry using K8s Secrets, in this
 case via the `kubectl create secret docker-registry` helper.

examples/demo-project/garden.yml

@@ -18,3 +18,10 @@ environments:
         namespace: demo-project-testing-${local.env.CIRCLE_BUILD_NUM || "default"}
         defaultHostname: demo-project-testing.dev-1.sys.garden
         buildMode: cluster-docker
+  - name: dev-2
+    providers:
+      - name: kubernetes
+        context: gke_garden-dev-200012_europe-west3-a_dev-2
+        namespace: ${local.env.USER || "default"}-demo-project
+        defaultHostname: ${local.env.USER || "default"}-demo-project.dev-1.sys.garden
+        buildMode: cluster-docker

examples/gradle/garden.yml

@@ -0,0 +1,54 @@
+kind: Project
+name: my-service
+environments:
+  - name: local
+    providers:
+      - name: local-kubernetes
+---
+kind: Module
+description: kafka module
+type: helm
+name: kafka
+chart: incubator/kafka
+values:
+  usePassword: false
+---
+kind: Module
+type: exec
+name: gradle
+description: gradle build
+build:
+  command:
+    - gradle
+    - clean
+    - test
+    - build
+---
+kind: Module
+name: my-service
+description: my-service container
+type: container
+build:
+  dependencies:
+    - name: gradle
+      # This is needed because builds for each module are staged separately, to avoid conflicts
+      copy:
+        - source: path/to/my.jar
+          target: path/to/my.jar
+include:
+  # List things you may need other than the JAR
+  - "other-source-files/**/*"
+services:
+  - name: my-service
+    ports:
+      - name: http
+        containerPort: 8080
+        servicePort: 80
+    dependencies:
+      - kafka
+      # - gradle    <-- Not needed, it's only a build dependency
+    env:
+      KAFKA_BROKER_ADDRESSES: kafka.my-service:9092
+      APPLICATION_ID_PREFIX: my-service
+      PRODUCER_TOPIC: data.object.persist
+      CONSUMER_TOPICS: raw.data

garden-service/src/commands/base.ts

@@ -223,10 +223,13 @@ export interface CommandResult<T = any> {
   errors?: GardenError[]
 }
 
-export interface PrepareParams<T extends Parameters = {}, U extends Parameters = {}> {
+export interface CommandParamsBase<T extends Parameters = {}, U extends Parameters = {}> {
   args: ParameterValues<T>
   opts: ParameterValues<GlobalOptions & U>
   log: LogEntry
+}
+
+export interface PrepareParams<T extends Parameters = {}, U extends Parameters = {}> extends CommandParamsBase<T, U> {
   headerLog: LogEntry
   footerLog: LogEntry
 }

garden-service/src/commands/commands.ts

@@ -28,6 +28,7 @@ import { ExecCommand } from "./exec"
 import { ServeCommand } from "./serve"
 import { OptionsCommand } from "./options"
 import { ConfigCommand } from "./config/config"
+import { PluginsCommand } from "./plugins"
 
 export const coreCommands: Command[] = [
   new BuildCommand(),
@@ -41,6 +42,7 @@ export const coreCommands: Command[] = [
   new LinkCommand(),
   new LogsCommand(),
   new OptionsCommand(),
+  new PluginsCommand(),
   new PublishCommand(),
   new RunCommand(),
   new ScanCommand(),