feat(core): pre-fetch provider tools and make tools a native feature

This adds the notion of tools to the plugin framework, and a command to
allow using and pre-fetching these tools.

The primary motivation was to allow pre-fetching the tools as part of
our container builds, as well as to make it easy for users to do the
same in their custom CI setups with Garden.

The `garden tools` command is also handy since it allows users to easily
use the exact same tools as Garden's providers fetch and use.

To pre-fetch the tools, we add a `garden util fetch-tools` command,
which fetches all tools for the configured providers in the current
project, or for all registered providers if the `--all` flag is set.

Notes:
- We now always fetch the docker binary on use, instead of checking for
  its existence locally. This avoids some code complexity and makes sure
  we always use the expected version. The downside should be mitigated
  by the pre-fetching in built containers, and the fetch-tools command.
- We really need to reduce the number of tools and versions we bundle
  before we release 0.12. The container image sizes are too large atm.

docs/reference/commands.md

@@ -86,6 +86,34 @@ Note: Currently only supports simple GET requests for HTTP/HTTPS ingresses.
 | -------- | -------- | ----------- |
   | `serviceAndPath` | Yes | The name of the service to call followed by the ingress path (e.g. my-container/somepath).
 
+### garden config analytics-enabled
+
+Update your preferences regarding analytics.
+
+To help us make Garden better, you can opt in to the collection of usage data.
+We make sure all the data collected is anonymized and stripped of sensitive
+information. We collect data about which commands are run, what tasks they trigger,
+which API calls are made to your local Garden server, as well as some info
+about the environment in which Garden runs.
+
+You will be asked if you want to opt-in when running Garden for the
+first time and you can use this command to update your preferences later.
+
+Examples:
+
+    garden config analytics-enabled true   # enable analytics
+    garden config analytics-enabled false  # disable analytics
+
+##### Usage
+
+    garden config analytics-enabled [enable] 
+
+##### Arguments
+
+| Argument | Required | Description |
+| -------- | -------- | ----------- |
+  | `enable` | No | Enable analytics. Defaults to "true"
+
 ### garden create project
 
 Create a new Garden project.
@@ -793,6 +821,56 @@ Examples:
   | `--force-build` |  | boolean | Force rebuild of module(s).
   | `--watch` | `-w` | boolean | Watch for changes in module(s) and auto-test.
 
+### garden tools
+
+Access tools included by providers.
+
+Run a tool defined by a provider in your project, downloading and extracting it if
+necessary. Run without arguments to get a list of all tools available.
+
+Run with the --get-path flag to just print the path to the binary or library
+directory (depending on the tool type). If the tool is a non-executable library, this
+flag is implicit.
+
+When multiple plugins provide a tool with the same name, you can choose a specific
+plugin/version by specifying <plugin name>.<tool name>, instead of just <tool name>.
+This is generally advisable when using this command in scripts, to avoid accidental
+conflicts.
+
+When there are name conflicts and a plugin name is not specified, the preference is
+for defined by configured providers in the current project (if applicable), and then
+alphabetical by plugin name.
+
+Examples:
+
+    # Run kubectl with <args>.
+    garden tools kubectl -- <args>
+
+    # Run the kubectl version defined specifically by the `kubernetes` plugin.
+    garden tools kubernetes.kubectl -- <args>
+
+    # Print the path to the kubernetes.kubectl tool to stdout, instead of running it.
+    garden tools kubernetes.kubectl --get-path
+
+    # List all available tools.
+    garden tools
+
+##### Usage
+
+    garden tools [tool] [options]
+
+##### Arguments
+
+| Argument | Required | Description |
+| -------- | -------- | ----------- |
+  | `tool` | No | The name of the tool to run.
+
+##### Options
+
+| Argument | Alias | Type | Description |
+| -------- | ----- | ---- | ----------- |
+  | `--get-path` |  | boolean | If specified, we print the path to the binary or library instead of running it.
+
 ### garden unlink source
 
 Unlink a previously linked remote source from its local directory.
@@ -904,41 +982,36 @@ Examples:
 
     garden update-remote all 
 
-### garden validate
+### garden util fetch-tools
 
-Check your garden configuration for errors.
+Pre-fetch plugin tools.
 
-Throws an error and exits with code 1 if something's not right in your garden.yml files.
+Pre-fetch all the available tools for the configured providers in the current
+project/environment, or all registered providers if the --all parameter is
+specified.
 
-##### Usage
+Examples:
 
-    garden validate 
+    garden util fetch-tools        # fetch for just the current project/env
+    garden util fetch-tools --all  # fetch for all registered providers
 
-### garden config analytics-enabled
+##### Usage
 
-Update your preferences regarding analytics.
+    garden util fetch-tools [options]
 
-To help us make Garden better, you can opt in to the collection of usage data.
-We make sure all the data collected is anonymized and stripped of sensitive
-information. We collect data about which commands are run, what tasks they trigger,
-which API calls are made to your local Garden server, as well as some info
-about the environment in which Garden runs.
-
-You will be asked if you want to opt-in when running Garden for the
-first time and you can use this command to update your preferences later.
+##### Options
 
-Examples:
+| Argument | Alias | Type | Description |
+| -------- | ----- | ---- | ----------- |
+  | `--all` |  | boolean | Fetch all tools for registered plugins, instead of just ones in the current env/project.
 
-    garden config analytics-enabled true   # enable analytics
-    garden config analytics-enabled false  # disable analytics
+### garden validate
 
-##### Usage
+Check your garden configuration for errors.
 
-    garden config analytics-enabled [enable] 
+Throws an error and exits with code 1 if something's not right in your garden.yml files.
 
-##### Arguments
+##### Usage
 
-| Argument | Required | Description |
-| -------- | -------- | ----------- |
-  | `enable` | No | Enable analytics. Defaults to &quot;true&quot;
+    garden validate 
 

garden-service/Dockerfile

@@ -11,17 +11,17 @@ RUN apk add --no-cache \
 
 WORKDIR /tmp
 
-RUN npm install [email protected].2 && node_modules/.bin/pkg-fetch node12 alpine x64
+RUN npm install [email protected].8 && node_modules/.bin/pkg-fetch node12 alpine x64
 
 ADD package.json /tmp/
 ADD package-lock.json /tmp/
 
 RUN npm install \
   && rm -rf /root/.npm/* \
-    /usr/lib/node_modules/npm/man/* \
-    /usr/lib/node_modules/npm/doc/* \
-    /usr/lib/node_modules/npm/html/* \
-    /usr/lib/node_modules/npm/scripts/*
+  /usr/lib/node_modules/npm/man/* \
+  /usr/lib/node_modules/npm/doc/* \
+  /usr/lib/node_modules/npm/html/* \
+  /usr/lib/node_modules/npm/scripts/*
 
 ADD bin/garden /tmp/bin/garden
 ADD bin/garden-debug /tmp/bin/garden-debug
@@ -56,6 +56,7 @@ WORKDIR /project
 
 RUN chmod +x /garden/garden \
   && ln -s /garden/garden /bin/garden \
-  && chmod +x /bin/garden
+  && chmod +x /bin/garden \
+  && cd /garden/static && garden util fetch-tools --all --logger-type=basic
 
 ENTRYPOINT ["/garden/garden"]

garden-service/buster.Dockerfile

@@ -26,6 +26,9 @@ ADD . /garden
 WORKDIR /project
 
 RUN ln -s /garden/garden /bin/garden \
-  && chmod +x /bin/garden
+  && chmod +x /bin/garden \
+  && cd /garden/static \
+  && git init \
+  && garden util fetch-tools --all --logger-type=basic
 
 ENTRYPOINT ["/garden/garden"]

garden-service/src/actions.ts

@@ -624,7 +624,7 @@ export class ActionRouter implements TypeGuard {
     const environmentStatuses: EnvironmentStatusMap = {}
 
     const providers = await this.garden.resolveProviders()
-    await Bluebird.each(providers, async (provider) => {
+    await Bluebird.each(Object.values(providers), async (provider) => {
       await this.cleanupEnvironment({ pluginName: provider.name, log: envLog })
       environmentStatuses[provider.name] = await this.getEnvironmentStatus({ pluginName: provider.name, log: envLog })
     })

garden-service/src/analytics/analytics.ts

@@ -19,7 +19,6 @@ import { Garden } from "../garden"
 import { AnalyticsType } from "./analytics-types"
 import dedent from "dedent"
 import { getGitHubUrl } from "../docs/common"
-import { InternalError } from "../exceptions"
 
 const API_KEY = process.env.ANALYTICS_DEV ? SEGMENT_DEV_API_KEY : SEGMENT_PROD_API_KEY
 
@@ -126,15 +125,12 @@ export class AnalyticsHandler {
   private ciName = ci.name
   private systemConfig: SystemInfo
   private isCI = ci.isCI
-  private sessionId: string
+  private sessionId: string | null
   private pendingEvents: Map<string, SegmentEvent>
   protected garden: Garden
   private projectMetadata: ProjectMetadata
 
   private constructor(garden: Garden, log: LogEntry) {
-    if (!garden.sessionId) {
-      throw new InternalError(`Garden instance with null sessionId passed to AnalyticsHandler constructor.`, {})
-    }
     this.segment = new segmentClient(API_KEY, { flushAt: 20, flushInterval: 300 })
     this.log = log
     this.garden = garden
@@ -247,7 +243,7 @@ export class AnalyticsHandler {
    * Used internally to check if a users has opted-in or not.
    */
   private analyticsEnabled(): boolean {
-    if (process.env.GARDEN_DISABLE_ANALYTICS === "true") {
+    if (!this.sessionId || process.env.GARDEN_DISABLE_ANALYTICS === "true") {
       return false
     }
     return this.analyticsConfig.optedIn || false
@@ -282,7 +278,7 @@ export class AnalyticsHandler {
       ciName: this.ciName,
       system: this.systemConfig,
       isCI: this.isCI,
-      sessionId: this.sessionId,
+      sessionId: this.sessionId!,
       projectMetadata: this.projectMetadata,
     }
   }

garden-service/src/cli/cli.ts

@@ -73,9 +73,9 @@ const GLOBAL_OPTIONS_GROUP_NAME = "Global options"
  * Used by commands that have noProject=true. That is, commands that need
  * to run outside of valid Garden projects.
  */
-class DummyGarden extends Garden {
+export class DummyGarden extends Garden {
   async resolveProviders() {
-    return []
+    return {}
   }
   async scanAndAddConfigs() {}
 }
@@ -307,7 +307,6 @@ export class GardenCli {
       // the screen the logs are printed.
       const headerLog = logger.placeholder()
       const log = logger.placeholder()
-      logger.info("")
       const footerLog = logger.placeholder()
 
       // Init event & log streaming.

garden-service/src/commands/commands.ts

@@ -32,17 +32,22 @@ import { ConfigCommand } from "./config/config"
 import { PluginsCommand } from "./plugins"
 import { LoginCommand } from "./login"
 import { LogOutCommand } from "./logout"
+import { ToolsCommand } from "./tools"
+import { UtilCommand } from "./util"
 
 export const coreCommands: Command[] = [
   new BuildCommand(),
   new CallCommand(),
+  new ConfigCommand(),
   new CreateCommand(),
   new DeleteCommand(),
   new DeployCommand(),
   new DevCommand(),
   new ExecCommand(),
   new GetCommand(),
   new LinkCommand(),
+  new LoginCommand(),
+  new LogOutCommand(),
   new LogsCommand(),
   new MigrateCommand(),
   new OptionsCommand(),
@@ -53,10 +58,9 @@ export const coreCommands: Command[] = [
   new ServeCommand(),
   new SetCommand(),
   new TestCommand(),
+  new ToolsCommand(),
   new UnlinkCommand(),
   new UpdateRemoteCommand(),
+  new UtilCommand(),
   new ValidateCommand(),
-  new ConfigCommand(),
-  new LoginCommand(),
-  new LogOutCommand(),
 ]

garden-service/src/commands/plugins.ts

@@ -127,7 +127,7 @@ async function listPlugins(garden: Garden, log: LogEntry, pluginsToList: string[
   log.info(dedent`
   ${chalk.white.bold("USAGE")}
 
-    garden ${chalk.yellow("[global options]")} ${chalk.blueBright("<command>")} ${chalk.white("[args ...]")}
+    garden ${chalk.yellow("[global options]")} ${chalk.blueBright("<command>")} -- ${chalk.white("[args ...]")}
 
   ${chalk.white.bold("PLUGIN COMMANDS")}
   `)