diff --git a/docs/reference/commands.md b/docs/reference/commands.md index ec1b4c80b2..13bf828e70 100644 --- a/docs/reference/commands.md +++ b/docs/reference/commands.md @@ -317,6 +317,7 @@ Examples: | Argument | Alias | Type | Description | | -------- | ----- | ---- | ----------- | | `--dir` | | path | Directory to place the project in (defaults to current directory). + | `--filename` | | string | Filename to place the project config in (defaults to project.garden.yml). | `--interactive` | `-i` | boolean | Set to false to disable interactive prompts. | `--name` | | string | Name of the project (defaults to current directory name). @@ -348,6 +349,7 @@ Examples: | Argument | Alias | Type | Description | | -------- | ----- | ---- | ----------- | | `--dir` | | path | Directory to place the module in (defaults to current directory). + | `--filename` | | string | Filename to place the module config in (defaults to garden.yml). | `--interactive` | `-i` | boolean | Set to false to disable interactive prompts. | `--name` | | string | Name of the module (defaults to current directory name). | `--type` | | string | The module type to create. Required if --interactive=false. diff --git a/docs/using-garden/configuration-overview.md b/docs/using-garden/configuration-overview.md index 8ade050037..3ccdcdb616 100644 --- a/docs/using-garden/configuration-overview.md +++ b/docs/using-garden/configuration-overview.md @@ -5,16 +5,14 @@ title: Configuration Overview # Configuration Overview -Garden is configured via `garden.yml` configuration files, which Garden collects and compiles into a +Garden is configured via `garden.yml` (or `*.garden.yml`) configuration files, which Garden collects and compiles into a [Stack Graph](../basics/stack-graph.md) of your project. -The [project configuration](./projects.md) `garden.yml` file should be located in the top-level directory of the -project's Git repository. +The [project configuration](./projects.md) file should be located in the top-level directory of the project's Git repository. We suggest naming it `project.garden.yml` for clarity, but you can also use `garden.yml` or any filename ending with `.garden.yml`. -In addition, each of the project's [modules](../reference/glossary.md#module)' `garden.yml` should be located in that -module's top-level directory. Modules define all the individual components of your project, including [services](./services.md), [tasks](./tasks.md) and [tests](./tests.md). +In addition, each of the project's [modules](../reference/glossary.md#module)' should be located in that module's top-level directory. Modules define all the individual components of your project, including [services](./services.md), [tasks](./tasks.md) and [tests](./tests.md). -Lastly, you can define [workflows](./workflows.md), to codify sequences of Garden commands and custom scripts. +Lastly, you can define [workflows](./workflows.md), to codify sequences of Garden commands and custom scripts. We suggest placing those in a `workflows.garden.yml` file in your project root. The other docs under the _Using Garden_ go into more details, and we highly recommend reading through all of them. diff --git a/docs/using-garden/modules.md b/docs/using-garden/modules.md index 2f6587ebb1..cdf9a543cb 100644 --- a/docs/using-garden/modules.md +++ b/docs/using-garden/modules.md @@ -26,11 +26,11 @@ tests: ## How it Works -A Garden project is usually split up into the project-level `garden.yml` file and several module-level configuration files: +A Garden project is usually split up into the project-level configuration file, and several module-level configuration files, each in the root directory of the respective module: ```console . -├── garden.yml +├── project.garden.yml ├── module-a │   ├── garden.yml │   └── ... @@ -42,8 +42,10 @@ A Garden project is usually split up into the project-level `garden.yml` file an    └── ... ``` +You can also choose any `*.garden.yml` filename for each module configuration file. For example, you might prefer to set the module name in the filename, e.g. `my-module.garden.yml` to make it easier to find in a large project. + {% hint style="info" %} -It's also possible to [define several modules in the same `garden.yml` file](#multiple-modules-in-the-same-directory) and/or in the same file as the the project-level configuration. +It's also possible to [define several modules in the same `garden.yml` file](#multiple-modules-in-the-same-directory) and/or in the same file as the the project-level configuration. If you only have a couple of modules, you might for example define them together in a single `modules.garden.yml` file. See [below](#multiple-modules-in-the-same-directory) for more details. {% endhint %} Modules must have a type. Different [module _types_](#module-types) behave in different ways. For example, the `container` module type corresponds to a Docker image, either built from a local Dockerfile or pulled from a remote repository. @@ -94,14 +96,11 @@ You can also use [.gardenignore files](./configuration-overview.md#ignore-files) ### Multiple modules in the same directory -Sometimes, it's useful to define several modules in the same `garden.yml` file. One common situation is where more than -one Dockerfile is in use (e.g. one for a development build and one for a production build). +Sometimes, it's useful to define several modules in the same `garden.yml` file. One common situation is where more than one Dockerfile is in use (e.g. one for a development build and one for a production build). You may only have a handful of modules, and it may be the cleanest approach to define all of them in a `modules.garden.yml` in your project root. -Another is when the dev configuration and the production configuration have different integration testing suites, -which may depend on different external services being available. +Another example is when the dev configuration and the production configuration have different integration testing suites, which may depend on different external services being available. -To do this, add a document separator (`---`) between the module definitions. Here's a simple (if a bit contrived) -example: +To do this, add a document separator (`---`) between the module definitions. Here's a simple (if a bit contrived) example: ```yaml kind: Module @@ -135,7 +134,9 @@ tests: - b-integration-testing-backend ``` -Note that you must use the `include` and/or `exclude` directives (described above) when module paths overlap. This is to help users steer away from subtle bugs that can occur when modules unintentionally consume source files from other modules. See the next section for details on including and excluding files. +{% hint style="warning" %} +Note that you **must** use the `include` and/or `exclude` directives (described above) when module paths overlap. This is to help users steer away from subtle bugs that can occur when modules unintentionally consume source files from other modules. See the next section for details on including and excluding files. +{% endhint %} ## Modules in the Stack Graph diff --git a/docs/using-garden/projects.md b/docs/using-garden/projects.md index 9a52fd290b..0b148bb41a 100644 --- a/docs/using-garden/projects.md +++ b/docs/using-garden/projects.md @@ -5,10 +5,10 @@ title: Projects # Projects -The first step to using Garden is to create a project. You can use the `garden create project` helper command, or manually create a `garden.yml` file in the root directory of your project: +The first step to using Garden is to create a project. You can use the `garden create project` helper command, or manually create a `project.garden.yml` file in the root directory of your project: ```yaml -# garden.yml - located in the top-level directory of your project +# project.garden.yml - located in the top-level directory of your project kind: Project name: my-project environments: @@ -18,17 +18,19 @@ providers: environments: ["local"] ``` +We suggest naming it `project.garden.yml` for clarity, but you can also use `garden.yml` or any filename ending with `.garden.yml`. + The helper command has the benefit of including all the possible fields you can configure, and their documentation, so you can quickly scan through the available options and uncomment as needed. ## How it Works -The top-level `garden.yml` file is where project-wide configuration takes place. This includes environment configurations and project variables. Most importantly, it's where you declare and configure the *providers* you want to use for your project ([see below](#providers)). +The top-level `project.garden.yml` file is where project-wide configuration takes place. This includes environment configurations and project variables. Most importantly, it's where you declare and configure the *providers* you want to use for your project ([see below](#providers)). Garden treats the directory containing the project configuration as the project's top-level directory. Garden commands that run in subdirectories of the project root are assumed to apply to that project, and commands above/outside a project root will fail—similarly to how Git uses the location of the repo's `.git` directory as the repo's root directory. ### Environments and namespaces -Every Garden command is run against one of the environments defined in the project-level `garden.yml` file. You can specify the environment with the `--env` flag or by setting a `defaultEnvironment`. Alternatively, Garden defaults to the first environment in the `garden.yml` file. +Every Garden command is run against one of the environments defined in the project-level configuration file. You can specify the environment with the `--env` flag or by setting a `defaultEnvironment`. Alternatively, Garden defaults to the first environment defined in your configuration. An environment can be partitioned using _namespaces_. A common use-case is to split a shared development or testing environment by namespace, between e.g. users or different branches of code. diff --git a/docs/using-garden/workflows.md b/docs/using-garden/workflows.md index 2df3ab6e24..4e6aefbb93 100644 --- a/docs/using-garden/workflows.md +++ b/docs/using-garden/workflows.md @@ -16,13 +16,14 @@ A sequence of commands executed in a workflow is also generally more efficent th Workflows are defined with a separate _kind_ of configuration file, with a list of _steps_: ```yaml +# workflows.garden.yml kind: Workflow name: my-workflow steps: - ... ``` -You can place your workflow definitions in your project root `garden.yml` file (with a `---` separator after the project configuration), or in a separate `garden.yml` in a different directory, e.g. in `workflows/garden.yml`. +We suggest making a `workflows.garden.yml` next to your project configuration in your project root. You can also place your workflow definitions in your project root `project.garden.yml`/`garden.yml` file (with a `---` separator after the project configuration). Each step in your workflow can either trigger Garden commands, or run custom scripts. The steps are executed in succession. If a step fails, the remainder of the workflow is aborted. diff --git a/garden-service/src/commands/base.ts b/garden-service/src/commands/base.ts index afa207de55..ec51a49e3b 100644 --- a/garden-service/src/commands/base.ts +++ b/garden-service/src/commands/base.ts @@ -141,7 +141,7 @@ export class StringsParameter extends Parameter { export class PathParameter extends Parameter { type = "path" - schema = joi.posixPath() + schema = joi.string() } export class PathsParameter extends Parameter { diff --git a/garden-service/src/commands/create/create-module.ts b/garden-service/src/commands/create/create-module.ts index 0328b81187..fd53eb43c9 100644 --- a/garden-service/src/commands/create/create-module.ts +++ b/garden-service/src/commands/create/create-module.ts @@ -18,11 +18,12 @@ import { PathParameter, BooleanParameter, StringOption, + StringParameter, } from "../base" import { printHeader } from "../../logger/util" -import { getConfigFilePath, isDirectory } from "../../util/fs" -import { loadConfig, findProjectConfig } from "../../config/base" -import { resolve, basename, relative } from "path" +import { isDirectory, defaultConfigFilename } from "../../util/fs" +import { loadConfigResources, findProjectConfig } from "../../config/base" +import { resolve, basename, relative, join } from "path" import { GardenBaseError, ParameterError } from "../../exceptions" import { getModuleTypes, getPluginBaseNames } from "../../plugins" import { addConfig } from "./helpers" @@ -46,6 +47,10 @@ const createModuleOpts = { help: "Directory to place the module in (defaults to current directory).", defaultValue: ".", }), + filename: new StringParameter({ + help: "Filename to place the module config in (defaults to garden.yml).", + defaultValue: defaultConfigFilename, + }), interactive: new BooleanParameter({ alias: "i", help: "Set to false to disable interactive prompts.", @@ -113,7 +118,7 @@ export class CreateModuleCommand extends Command c.kind === "Module" && c.name === name).length > 0) { throw new CreateError( diff --git a/garden-service/src/commands/create/create-project.ts b/garden-service/src/commands/create/create-project.ts index dba7ea3dbf..be1507c275 100644 --- a/garden-service/src/commands/create/create-project.ts +++ b/garden-service/src/commands/create/create-project.ts @@ -18,11 +18,12 @@ import { PathParameter, BooleanParameter, StringOption, + StringParameter, } from "../base" import { printHeader } from "../../logger/util" -import { getConfigFilePath, isDirectory } from "../../util/fs" -import { loadConfig } from "../../config/base" -import { resolve, basename, relative } from "path" +import { isDirectory } from "../../util/fs" +import { loadConfigResources } from "../../config/base" +import { resolve, basename, relative, join } from "path" import { GardenBaseError, ParameterError } from "../../exceptions" import { renderProjectConfigReference } from "../../docs/config" import { addConfig } from "./helpers" @@ -36,12 +37,18 @@ const defaultIgnorefile = dedent` # For more info, see https://docs.garden.io/using-garden/configuration-overview#including-excluding-files-and-directories ` +export const defaultProjectConfigFilename = "project.garden.yml" + const createProjectArgs = {} const createProjectOpts = { dir: new PathParameter({ help: "Directory to place the project in (defaults to current directory).", defaultValue: ".", }), + filename: new StringParameter({ + help: "Filename to place the project config in (defaults to project.garden.yml).", + defaultValue: defaultProjectConfigFilename, + }), interactive: new BooleanParameter({ alias: "i", help: "Set to false to disable interactive prompts.", @@ -107,11 +114,11 @@ export class CreateProjectCommand extends Command c.kind === "Project").length > 0) { throw new CreateError(`A Garden project already exists in ${configPath}`, { configDir, configPath }) diff --git a/garden-service/src/commands/get/get-debug-info.ts b/garden-service/src/commands/get/get-debug-info.ts index b0a4dc7bcb..abe85feca2 100644 --- a/garden-service/src/commands/get/get-debug-info.ts +++ b/garden-service/src/commands/get/get-debug-info.ts @@ -13,8 +13,7 @@ import { getPackageVersion, exec, safeDumpYaml } from "../../util/util" import { platform, release } from "os" import { join, relative, basename, dirname } from "path" import { LogEntry } from "../../logger/log-entry" -import { deline } from "../../util/string" -import { findConfigPathsInPath, getConfigFilePath, defaultDotIgnoreFiles } from "../../util/fs" +import { findConfigPathsInPath, defaultDotIgnoreFiles } from "../../util/fs" import { ERROR_LOG_FILENAME } from "../../constants" import dedent = require("dedent") import { Garden } from "../../garden" @@ -40,12 +39,10 @@ export const PROVIDER_INFO_FILENAME_NO_EXT = "info" */ export async function collectBasicDebugInfo(root: string, gardenDirPath: string, log: LogEntry) { // Find project definition - const config = await findProjectConfig(root, true) - if (!config) { + const projectConfig = await findProjectConfig(root, true) + if (!projectConfig) { throw new ValidationError( - deline` - Couldn't find a garden.yml with a project definition. - Please run this command from the root of your Garden project.`, + "Couldn't find a Project definition. Please run this command from the root of your Garden project.", {} ) } @@ -56,18 +53,19 @@ export async function collectBasicDebugInfo(root: string, gardenDirPath: string, await ensureDir(tempPath) // Copy project definition in tmp folder - const projectConfigFilePath = await getConfigFilePath(root) + const projectConfigFilePath = projectConfig.configPath! const projectConfigFilename = basename(projectConfigFilePath) await copy(projectConfigFilePath, join(tempPath, projectConfigFilename)) + // Check if error logs exist and copy it over if it does if (await pathExists(join(root, ERROR_LOG_FILENAME))) { await copy(join(root, ERROR_LOG_FILENAME), join(tempPath, ERROR_LOG_FILENAME)) } // Find all services paths - const vcs = new GitHandler(root, config.dotIgnoreFiles || defaultDotIgnoreFiles) - const include = config.modules && config.modules.include - const exclude = config.modules && config.modules.exclude + const vcs = new GitHandler(root, projectConfig.dotIgnoreFiles || defaultDotIgnoreFiles) + const include = projectConfig.modules && projectConfig.modules.include + const exclude = projectConfig.modules && projectConfig.modules.exclude const paths = await findConfigPathsInPath({ vcs, dir: root, include, exclude, log }) // Copy all the service configuration files @@ -80,14 +78,13 @@ export async function collectBasicDebugInfo(root: string, gardenDirPath: string, }) const tempServicePath = join(tempPath, relative(root, servicePath)) await ensureDir(tempServicePath) - const moduleConfigFilePath = await getConfigFilePath(servicePath) - const moduleConfigFilename = basename(moduleConfigFilePath) + const moduleConfigFilename = basename(configPath) const gardenLog = gardenPathLog.info({ section: moduleConfigFilename, msg: "collecting garden.yml", status: "active", }) - await copy(moduleConfigFilePath, join(tempServicePath, moduleConfigFilename)) + await copy(configPath, join(tempServicePath, moduleConfigFilename)) gardenLog.setSuccess({ msg: chalk.green(`Done (took ${log.getDuration(1)} sec)`), append: true }) // Check if error logs exist and copy them over if they do if (await pathExists(join(servicePath, ERROR_LOG_FILENAME))) { diff --git a/garden-service/src/commands/migrate.ts b/garden-service/src/commands/migrate.ts index 8493335fdf..7690838e37 100644 --- a/garden-service/src/commands/migrate.ts +++ b/garden-service/src/commands/migrate.ts @@ -11,14 +11,14 @@ import { dedent } from "../util/string" import { readFile, writeFile } from "fs-extra" import { cloneDeep, isEqual } from "lodash" import { ConfigurationError, RuntimeError } from "../exceptions" -import { resolve, parse } from "path" -import { findConfigPathsInPath, getConfigFilePath } from "../util/fs" +import { resolve } from "path" +import { findConfigPathsInPath } from "../util/fs" import { GitHandler } from "../vcs/git" import { DEFAULT_GARDEN_DIR_NAME } from "../constants" import { exec, safeDumpYaml } from "../util/util" import { LoggerType } from "../logger/logger" import Bluebird from "bluebird" -import { loadAndValidateYaml } from "../config/base" +import { loadAndValidateYaml, findProjectConfig } from "../config/base" const migrateOpts = { write: new BooleanParameter({ help: "Update the `garden.yml` in place." }), @@ -71,13 +71,16 @@ export class MigrateCommand extends Command { async action({ log, args, opts }: CommandParams): Promise> { // opts.root defaults to current directory - const root = await findRoot(opts.root) - if (!root) { + const projectConfig = await findProjectConfig(opts.root, true) + + if (!projectConfig) { throw new ConfigurationError(`Not a project directory (or any of the parent directories): ${opts.root}`, { root: opts.root, }) } + const root = projectConfig.path + const updatedConfigs: { path: string; specs: any[] }[] = [] let configPaths: string[] = [] @@ -169,29 +172,6 @@ export function dumpSpec(specs: any[]) { return specs.map((spec) => safeDumpYaml(spec)).join("\n---\n\n") } -/** - * Recursively search for the project root by checking if the path has a project level `garden.yml` file - */ -async function findRoot(path: string): Promise { - const configFilePath = await getConfigFilePath(path) - let isProjectRoot = false - try { - const rawSpecs = await readYaml(configFilePath) - isProjectRoot = rawSpecs.find((spec) => !!spec.project || spec.kind === "Project") - } catch (err) { - // no op - } - if (isProjectRoot) { - return path - } - - // We're at the file system root and no project file was found - if (parse(path).root) { - return null - } - return findRoot(resolve(path, "..")) -} - /** * Read the contents of a YAML file and dump to JSON */ diff --git a/garden-service/src/config/base.ts b/garden-service/src/config/base.ts index 810556eef5..b635fe2577 100644 --- a/garden-service/src/config/base.ts +++ b/garden-service/src/config/base.ts @@ -6,18 +6,19 @@ * file, You can obtain one at http://mozilla.org/MPL/2.0/. */ -import { sep, resolve, relative, basename, dirname } from "path" +import { sep, resolve, relative, basename, dirname, join } from "path" import yaml from "js-yaml" import yamlLint from "yaml-lint" import { readFile } from "fs-extra" -import { omit, isPlainObject, find, isArray } from "lodash" +import { omit, isPlainObject, isArray } from "lodash" import { ModuleResource, coreModuleSpecSchema, baseModuleSchemaKeys, BuildDependencyConfig } from "./module" -import { ConfigurationError } from "../exceptions" +import { ConfigurationError, FilesystemError } from "../exceptions" import { DEFAULT_API_VERSION } from "../constants" import { ProjectResource } from "../config/project" -import { getConfigFilePath } from "../util/fs" import { validateWithPath } from "./validation" import { WorkflowResource } from "./workflow" +import { listDirectory } from "../util/fs" +import { isConfigFilename } from "../util/fs" export interface GardenResource { apiVersion: string @@ -51,31 +52,27 @@ export async function loadAndValidateYaml(content: string, path: string): Promis } } -export async function loadConfig(projectRoot: string, path: string): Promise { - const configPath = await getConfigFilePath(path) +export async function loadConfigResources( + projectRoot: string, + configPath: string, + allowInvalid = false +): Promise { let fileData: Buffer - // loadConfig returns undefined if config file is not found in the given directory try { fileData = await readFile(configPath) } catch (err) { - return [] + throw new FilesystemError(`Could not find configuration file at ${configPath}`, { projectRoot, configPath }) } - let rawSpecs = await loadAndValidateYaml(fileData.toString(), path) + let rawSpecs = await loadAndValidateYaml(fileData.toString(), configPath) // Ignore empty resources rawSpecs = rawSpecs.filter(Boolean) - const resources: GardenResource[] = rawSpecs.map((s) => prepareResource(s, path, configPath, projectRoot)) - - const projectSpecs = resources.filter((s) => s.kind === "Project") - - if (projectSpecs.length > 1) { - throw new ConfigurationError(`Multiple project declarations in ${path}`, { - projectSpecs, - }) - } + const resources = ( + rawSpecs.map((s) => prepareResource({ spec: s, configPath, projectRoot, allowInvalid })).filter(Boolean) + ) return resources } @@ -85,23 +82,35 @@ export type ConfigKind = "Module" | "Workflow" | "Project" /** * Each YAML document in a garden.yml file defines a project, a module or a workflow. */ -function prepareResource(spec: any, path: string, configPath: string, projectRoot: string): GardenResource { +function prepareResource({ + spec, + configPath, + projectRoot, + allowInvalid = false, +}: { + spec: any + configPath: string + projectRoot: string + allowInvalid?: boolean +}): GardenResource | null { if (!isPlainObject(spec)) { - throw new ConfigurationError(`Invalid configuration found in ${path}`, { + throw new ConfigurationError(`Invalid configuration found in ${configPath}`, { spec, - path, + configPath, }) } const kind = spec.kind - const relPath = `${relative(projectRoot, path)}/garden.yml` + const relPath = relative(projectRoot, configPath) if (kind === "Project") { - return prepareProjectConfig(spec, path, configPath) + return prepareProjectConfig(spec, configPath) } else if (kind === "Module") { - return prepareModuleResource(spec, path, configPath, projectRoot) + return prepareModuleResource(spec, configPath, projectRoot) } else if (kind === "Workflow") { - return prepareWorkflowResource(spec, path, configPath) + return prepareWorkflowResource(spec, configPath) + } else if (allowInvalid) { + return spec } else if (!kind) { throw new ConfigurationError(`Missing \`kind\` field in config at ${relPath}`, { kind, @@ -115,24 +124,19 @@ function prepareResource(spec: any, path: string, configPath: string, projectRoo } } -function prepareProjectConfig(spec: any, path: string, configPath: string): ProjectResource { +function prepareProjectConfig(spec: any, configPath: string): ProjectResource { if (!spec.apiVersion) { spec.apiVersion = DEFAULT_API_VERSION } spec.kind = "Project" - spec.path = path + spec.path = dirname(configPath) spec.configPath = configPath return spec } -export function prepareModuleResource( - spec: any, - path: string, - configPath: string, - projectRoot: string -): ModuleResource { +export function prepareModuleResource(spec: any, configPath: string, projectRoot: string): ModuleResource { /** * We allow specifying modules by name only as a shorthand: * dependencies: @@ -160,7 +164,7 @@ export function prepareModuleResource( exclude: spec.exclude, name: spec.name, outputs: {}, - path, + path: dirname(configPath), repositoryUrl: spec.repositoryUrl, serviceConfigs: [], spec: { @@ -184,13 +188,13 @@ export function prepareModuleResource( return config } -export function prepareWorkflowResource(spec: any, path: string, configPath: string): WorkflowResource { +export function prepareWorkflowResource(spec: any, configPath: string): WorkflowResource { if (!spec.apiVersion) { spec.apiVersion = DEFAULT_API_VERSION } spec.kind = "Workflow" - spec.path = path + spec.path = dirname(configPath) spec.configPath = configPath return spec @@ -198,20 +202,25 @@ export function prepareWorkflowResource(spec: any, path: string, configPath: str export async function findProjectConfig(path: string, allowInvalid = false): Promise { let sepCount = path.split(sep).length - 1 + for (let i = 0; i < sepCount; i++) { - try { - const resources = await loadConfig(path, path) - const projectResource = find(resources, (r) => r.kind === "Project") - if (projectResource) { - return projectResource - } - } catch (err) { - if (!allowInvalid) { - throw err + const configFiles = (await listDirectory(path, { recursive: false })).filter(isConfigFilename) + + for (const configFile of configFiles) { + const resources = await loadConfigResources(path, join(path, configFile), allowInvalid) + + const projectSpecs = resources.filter((s) => s.kind === "Project") + + if (projectSpecs.length > 1 && !allowInvalid) { + throw new ConfigurationError(`Multiple project declarations found in ${path}`, { + projectSpecs, + }) + } else if (projectSpecs.length > 0) { + return projectSpecs[0] } - } finally { - path = resolve(path, "..") } + + path = resolve(path, "..") } return diff --git a/garden-service/src/garden.ts b/garden-service/src/garden.ts index 98d4001bb2..8824b57b37 100644 --- a/garden-service/src/garden.ts +++ b/garden-service/src/garden.ts @@ -11,7 +11,7 @@ import chalk from "chalk" import { ensureDir } from "fs-extra" import dedent from "dedent" import { platform, arch } from "os" -import { parse, relative, resolve, dirname } from "path" +import { parse, relative, resolve, join } from "path" import { flatten, isString, sortBy, fromPairs, keyBy, mapValues, omit, cloneDeep } from "lodash" const AsyncLock = require("async-lock") @@ -36,7 +36,7 @@ import { ConfigGraph } from "./config-graph" import { TaskGraph, GraphResults, ProcessTasksOpts } from "./task-graph" import { getLogger } from "./logger/logger" import { PluginActionHandlers, GardenPlugin } from "./types/plugin/plugin" -import { loadConfig, findProjectConfig, prepareModuleResource } from "./config/base" +import { loadConfigResources, findProjectConfig, prepareModuleResource } from "./config/base" import { DeepPrimitiveMap, StringMap } from "./config/common" import { validateSchema } from "./config/validation" import { BaseTask } from "./tasks/base" @@ -53,11 +53,11 @@ import { EventBus } from "./events" import { Watcher } from "./watch" import { findConfigPathsInPath, - getConfigFilePath, getWorkingCopyId, fixedExcludes, detectModuleOverlap, ModuleOverlap, + defaultConfigFilename, } from "./util/fs" import { Provider, @@ -618,7 +618,7 @@ export class Garden { Bluebird.map(provider.moduleConfigs, async (moduleConfig) => { // Make sure module and all nested entities are scoped to the plugin moduleConfig.plugin = provider.name - return this.addModule(moduleConfig) + return this.addModuleConfig(moduleConfig) }) ) @@ -795,7 +795,11 @@ export class Garden { // Resolve modules from specs and add to the list await Bluebird.map(addModules || [], async (spec) => { const path = spec.path || this.projectRoot - const moduleConfig = prepareModuleResource(spec, path, path, this.projectRoot) + const moduleConfig = prepareModuleResource(spec, join(path, defaultConfigFilename), this.projectRoot) + + // There is no actual config file for plugin modules (which the prepare function assumes) + delete moduleConfig.configPath + const resolvedConfig = await resolveModuleConfig(this, moduleConfig, { configContext }) resolvedModules.push(await moduleFromConfig(this, resolvedConfig, resolvedModules)) graph = undefined @@ -974,7 +978,7 @@ export class Garden { const rawWorkflowConfigs: WorkflowConfig[] = [] await Bluebird.map(configPaths, async (path) => { - const configs = await this.loadConfigs(dirname(path)) + const configs = await this.loadResources(path) if (configs) { const moduleConfigs = configs.filter((c) => c.kind === "Module") const workflowConfigs = configs.filter((c) => c.kind === "Workflow") @@ -987,7 +991,7 @@ export class Garden { throwOnMissingSecretKeys(Object.fromEntries(rawWorkflowConfigs.map((c) => [c.name, c])), this.secrets, "Workflow") await Bluebird.all([ - Bluebird.map(rawModuleConfigs, async (config) => this.addModule(config)), + Bluebird.map(rawModuleConfigs, async (config) => this.addModuleConfig(config)), Bluebird.map(rawWorkflowConfigs, async (config) => this.addWorkflow(config)), ]) @@ -1007,15 +1011,14 @@ export class Garden { /** * Add a module config to the context, after validating and calling the appropriate configure plugin handler. */ - private async addModule(config: ModuleConfig) { + private async addModuleConfig(config: ModuleConfig) { const key = getModuleKey(config.name, config.plugin) this.log.silly(`Adding module ${key}`) + const existing = this.moduleConfigs[key] - if (this.moduleConfigs[key]) { - const paths = [this.moduleConfigs[key].path, config.path] - const [pathA, pathB] = ( - await Bluebird.map(paths, async (path) => relative(this.projectRoot, await getConfigFilePath(path))) - ).sort() + if (existing) { + const paths = [existing.configPath || existing.path, config.configPath || config.path] + const [pathA, pathB] = paths.map((path) => relative(this.projectRoot, path)).sort() throw new ConfigurationError(`Module ${key} is declared multiple times (in '${pathA}' and '${pathB}')`, { pathA, @@ -1035,11 +1038,11 @@ export class Garden { const key = config.name this.log.silly(`Adding workflow ${key}`) - if (this.workflowConfigs[key]) { - const paths = [this.workflowConfigs[key].path, config.path] - const [pathA, pathB] = ( - await Bluebird.map(paths, async (path) => relative(this.projectRoot, await getConfigFilePath(path))) - ).sort() + const existing = this.workflowConfigs[key] + + if (existing) { + const paths = [existing.configPath || existing.path, config.path] + const [pathA, pathB] = paths.map((path) => relative(this.projectRoot, path)).sort() throw new ConfigurationError(`Workflow ${key} is declared multiple times (in '${pathA}' and '${pathB}')`, { pathA, @@ -1051,16 +1054,16 @@ export class Garden { } /** - * Load a module and/or a workflow from the specified directory and return the configs, + * Load a module and/or a workflow from the specified config file path and return the configs, * or null if no module or workflow is found. * - * @param path Directory containing the module + * @param configPath Path to a garden config file */ - private async loadConfigs(path: string): Promise<(ModuleResource | WorkflowResource)[]> { - path = resolve(this.projectRoot, path) - this.log.silly(`Load module and workflow configs from ${path}`) - const resources = await loadConfig(this.projectRoot, path) - this.log.silly(`Loaded module and workflow configs from ${path}`) + private async loadResources(configPath: string): Promise<(ModuleResource | WorkflowResource)[]> { + configPath = resolve(this.projectRoot, configPath) + this.log.silly(`Load module and workflow configs from ${configPath}`) + const resources = await loadConfigResources(this.projectRoot, configPath) + this.log.silly(`Loaded module and workflow configs from ${configPath}`) return <(ModuleResource | WorkflowResource)[]>resources.filter((r) => r.kind === "Module" || r.kind === "Workflow") } diff --git a/garden-service/src/types/module.ts b/garden-service/src/types/module.ts index 5b42b365d9..9b06a684d3 100644 --- a/garden-service/src/types/module.ts +++ b/garden-service/src/types/module.ts @@ -13,7 +13,6 @@ import { ModuleVersion, moduleVersionSchema } from "../vcs/vcs" import { pathToCacheContext } from "../cache" import { Garden } from "../garden" import { joiArray, joiIdentifier, joiIdentifierMap, joi } from "../config/common" -import { getConfigFilePath } from "../util/fs" import { getModuleTypeBases } from "../plugins" import { ModuleType } from "./plugin/plugin" @@ -29,7 +28,6 @@ export interface Module { buildPath: string buildMetadataPath: string - configPath: string needsBuild: boolean version: ModuleVersion @@ -59,10 +57,7 @@ export const moduleSchema = () => compatibleTypes: joiArray(joiIdentifier()) .required() .description("A list of types that this module is compatible with (i.e. the module type itself + all bases)."), - configPath: joi - .string() - .required() - .description("The path to the module config file."), + configPath: joi.string().description("The path to the module config file, if applicable."), version: moduleVersionSchema().required(), buildDependencies: joiIdentifierMap(joi.link("...")) .required() @@ -100,7 +95,6 @@ export async function moduleFromConfig( config: ModuleConfig, buildDependencies: Module[] ): Promise { - const configPath = await getConfigFilePath(config.path) const version = await garden.resolveVersion(config, config.build.dependencies) const moduleTypes = await garden.getModuleTypes() const compatibleTypes = [config.type, ...getModuleTypeBases(moduleTypes[config.type], moduleTypes).map((t) => t.name)] @@ -110,7 +104,6 @@ export async function moduleFromConfig( buildPath: await garden.buildDir.buildPath(config), buildMetadataPath: await garden.buildDir.buildMetadataPath(config.name), - configPath, version, needsBuild: moduleNeedsBuild(config, moduleTypes[config.type]), diff --git a/garden-service/src/util/fs.ts b/garden-service/src/util/fs.ts index 1fcb9a7efd..c12c29bb96 100644 --- a/garden-service/src/util/fs.ts +++ b/garden-service/src/util/fs.ts @@ -9,12 +9,11 @@ import klaw = require("klaw") import glob from "glob" import _spawn from "cross-spawn" -import Bluebird from "bluebird" import { pathExists, readFile, writeFile, lstat } from "fs-extra" import minimatch = require("minimatch") import { some } from "lodash" import { join, basename, win32, posix } from "path" -import { ValidationError, FilesystemError } from "../exceptions" +import { FilesystemError } from "../exceptions" import { platform } from "os" import { VcsHandler } from "../vcs/vcs" import { LogEntry } from "../logger/log-entry" @@ -22,7 +21,7 @@ import { ModuleConfig } from "../config/module" import pathIsInside from "path-is-inside" import { uuidv4 } from "./util" -const VALID_CONFIG_FILENAMES = ["garden.yml", "garden.yaml"] +export const defaultConfigFilename = "garden.yml" const metadataFilename = "metadata.json" export const defaultDotIgnoreFiles = [".gardenignore"] export const fixedExcludes = [".git", ".gitmodules", ".garden/**/*", "debug-info*/**"] @@ -91,36 +90,16 @@ export function detectModuleOverlap(moduleConfigs: ModuleConfig[]): ModuleOverla return overlaps } -/** - * Returns the expected full path to the Garden config filename. - * - * If a valid config filename is found at the given path, it returns the full path to it. - * If no config file is found, it returns the path joined with the first value from the VALID_CONFIG_FILENAMES list. - * (The check for whether or not a project or a module has a valid config file at all is handled elsewehere.) - * - * Throws an error if there are more than one valid config filenames at the given path. - */ -export async function getConfigFilePath(path: string) { - const configFilePaths = await Bluebird.map(VALID_CONFIG_FILENAMES, async (filename) => { - const configFilePath = join(path, filename) - return (await pathExists(configFilePath)) ? configFilePath : undefined - }).filter(Boolean) - - if (configFilePaths.length > 1) { - throw new ValidationError(`Found more than one Garden config file at ${path}.`, { - path, - configFilenames: configFilePaths.map((filePath) => basename(filePath || "")).join(", "), - }) - } - - return configFilePaths[0] || join(path, VALID_CONFIG_FILENAMES[0]) -} - /** * Helper function to check whether a given filename is a valid Garden config filename */ export function isConfigFilename(filename: string) { - return VALID_CONFIG_FILENAMES.includes(filename) + return ( + filename === "garden.yml" || + filename === "garden.yaml" || + filename.endsWith(".garden.yml") || + filename.endsWith(".garden.yaml") + ) } export async function getChildDirNames(parentDir: string): Promise { diff --git a/garden-service/test/data/test-projects/custom-config-names/module-a/.garden-version b/garden-service/test/data/test-projects/custom-config-names/module-a/.garden-version new file mode 100644 index 0000000000..eb45f3a14b --- /dev/null +++ b/garden-service/test/data/test-projects/custom-config-names/module-a/.garden-version @@ -0,0 +1,3 @@ +{ + "contentHash": "1234567890" +} diff --git a/garden-service/test/data/test-projects/custom-config-names/module-a/garden.yml b/garden-service/test/data/test-projects/custom-config-names/module-a/garden.yml new file mode 100644 index 0000000000..af88619a40 --- /dev/null +++ b/garden-service/test/data/test-projects/custom-config-names/module-a/garden.yml @@ -0,0 +1,17 @@ +kind: Module +name: module-a +type: test +services: + - name: service-a +build: + command: [echo, A] +tests: + - name: unit + command: [echo, OK] + - name: integration + command: [echo, OK] + dependencies: + - service-a +tasks: + - name: task-a + command: [echo, OK] diff --git a/garden-service/test/data/test-projects/custom-config-names/module-b/module-b.garden.yaml b/garden-service/test/data/test-projects/custom-config-names/module-b/module-b.garden.yaml new file mode 100644 index 0000000000..f09543ae96 --- /dev/null +++ b/garden-service/test/data/test-projects/custom-config-names/module-b/module-b.garden.yaml @@ -0,0 +1,17 @@ +kind: Module +name: module-b +type: test +services: + - name: service-b + dependencies: + - service-a +build: + command: [echo, B] + dependencies: + - module-a +tests: + - name: unit + command: [echo, OK] +tasks: + - name: task-b + command: [echo, OK] diff --git a/garden-service/test/data/test-projects/custom-config-names/project.garden.yml b/garden-service/test/data/test-projects/custom-config-names/project.garden.yml new file mode 100644 index 0000000000..ab8c0c147e --- /dev/null +++ b/garden-service/test/data/test-projects/custom-config-names/project.garden.yml @@ -0,0 +1,16 @@ +kind: Project +name: test-project-a +id: test-project-id +environments: + - name: local + - name: other +providers: + - name: test-plugin + environments: [local] + - name: test-plugin-b + environments: [local] +variables: + some: variable +outputs: + - name: taskName + value: task-a diff --git a/garden-service/test/data/test-projects/custom-config-names/workflows.garden.yml b/garden-service/test/data/test-projects/custom-config-names/workflows.garden.yml new file mode 100644 index 0000000000..3dd430e091 --- /dev/null +++ b/garden-service/test/data/test-projects/custom-config-names/workflows.garden.yml @@ -0,0 +1,9 @@ +kind: Workflow +name: workflow-a +steps: + - script: echo +--- +kind: Workflow +name: workflow-b +steps: + - script: echo diff --git a/garden-service/test/unit/src/build-dir.ts b/garden-service/test/unit/src/build-dir.ts index 990ce62588..04191d17d3 100644 --- a/garden-service/test/unit/src/build-dir.ts +++ b/garden-service/test/unit/src/build-dir.ts @@ -12,8 +12,8 @@ import { pathExists, readdir, createFile } from "fs-extra" import { expect } from "chai" import { BuildTask } from "../../../src/tasks/build" import { makeTestGarden, dataDir, expectError, getDataDir, TestGarden } from "../../helpers" -import { getConfigFilePath } from "../../../src/util/fs" import { BuildDir } from "../../../src/build-dir" +import { defaultConfigFilename } from "../../../src/util/fs" /* Module dependency diagram for build-dir test project @@ -139,12 +139,12 @@ describe("BuildDir", () => { const graph = await garden.getConfigGraph(garden.log) const moduleA = graph.getModule("module-a") - moduleA.version.files = [await getConfigFilePath(moduleA.path)] + moduleA.version.files = [join(moduleA.path, defaultConfigFilename)] await garden.buildDir.syncFromSrc(moduleA, garden.log) const buildDirA = await garden.buildDir.buildPath(moduleA) - expect(await pathExists(await getConfigFilePath(buildDirA))).to.eql(true) + expect(await pathExists(join(buildDirA, defaultConfigFilename))).to.eql(true) expect(await pathExists(join(buildDirA, "some-dir", "some-file"))).to.eql(false) }) @@ -157,7 +157,7 @@ describe("BuildDir", () => { await createFile(deleteMe) - moduleA.version.files = [await getConfigFilePath(moduleA.path)] + moduleA.version.files = [join(moduleA.path, defaultConfigFilename)] await garden.buildDir.syncFromSrc(moduleA, garden.log) diff --git a/garden-service/test/unit/src/commands/create/create-module.ts b/garden-service/test/unit/src/commands/create/create-module.ts index 5a362d4ab4..d0d80b1bf9 100644 --- a/garden-service/test/unit/src/commands/create/create-module.ts +++ b/garden-service/test/unit/src/commands/create/create-module.ts @@ -11,7 +11,7 @@ import { withDefaultGlobalOpts, TempDirectory, makeTempDir, expectError } from " import { CreateModuleCommand, getModuleTypeSuggestions } from "../../../../../src/commands/create/create-module" import { makeDummyGarden } from "../../../../../src/cli/cli" import { Garden } from "../../../../../src/garden" -import { basename, join } from "path" +import { join } from "path" import { pathExists, readFile, writeFile, mkdirp } from "fs-extra" import { safeLoadAll } from "js-yaml" import { exec, safeDumpYaml } from "../../../../../src/util/util" @@ -19,6 +19,7 @@ import stripAnsi = require("strip-ansi") import { getModuleTypes } from "../../../../../src/plugins" import { supportedPlugins } from "../../../../../src/plugins/plugins" import inquirer = require("inquirer") +import { defaultConfigFilename } from "../../../../../src/util/fs" describe("CreateModuleCommand", () => { const command = new CreateModuleCommand() @@ -45,11 +46,17 @@ describe("CreateModuleCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir, interactive: false, name: undefined, type: "exec" }), + opts: withDefaultGlobalOpts({ + dir, + interactive: false, + name: undefined, + type: "exec", + filename: defaultConfigFilename, + }), }) const { name, configPath } = result! - expect(name).to.equal(basename("test")) + expect(name).to.equal("test") expect(configPath).to.equal(join(dir, "garden.yml")) expect(await pathExists(configPath)).to.be.true @@ -64,6 +71,27 @@ describe("CreateModuleCommand", () => { ]) }) + it("should allow overriding the default generated filename", async () => { + const { result } = await command.action({ + garden, + footerLog: garden.log, + headerLog: garden.log, + log: garden.log, + args: {}, + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: "test", + type: "exec", + filename: "custom.garden.yml", + }), + }) + const { configPath } = result! + + expect(configPath).to.equal(join(tmp.path, "custom.garden.yml")) + expect(await pathExists(configPath)).to.be.true + }) + it("should optionally set a module name", async () => { const { result } = await command.action({ garden, @@ -71,7 +99,13 @@ describe("CreateModuleCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir: tmp.path, interactive: false, name: "test", type: "exec" }), + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: "test", + type: "exec", + filename: defaultConfigFilename, + }), }) const { name, configPath } = result! @@ -100,7 +134,13 @@ describe("CreateModuleCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir: tmp.path, interactive: false, name: "test", type: "exec" }), + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: "test", + type: "exec", + filename: defaultConfigFilename, + }), }) const { name, configPath } = result! @@ -132,7 +172,13 @@ describe("CreateModuleCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir: tmp.path, interactive: false, name: "test", type: "exec" }), + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: "test", + type: "exec", + filename: defaultConfigFilename, + }), }), (err) => expect(stripAnsi(err.message)).to.equal("A Garden module named test already exists in " + configPath) ) @@ -148,7 +194,13 @@ describe("CreateModuleCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir, interactive: false, name: "test", type: "exec" }), + opts: withDefaultGlobalOpts({ + dir, + interactive: false, + name: "test", + type: "exec", + filename: defaultConfigFilename, + }), }), (err) => expect(err.message).to.equal(`Path ${dir} does not exist`) ) @@ -163,7 +215,13 @@ describe("CreateModuleCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir: tmp.path, interactive: false, name: undefined, type: "foo" }), + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: undefined, + type: "foo", + filename: defaultConfigFilename, + }), }), (err) => expect(stripAnsi(err.message)).to.equal("Could not find module type foo") ) diff --git a/garden-service/test/unit/src/commands/create/create-project.ts b/garden-service/test/unit/src/commands/create/create-project.ts index 1dfd22974f..b7d517ed31 100644 --- a/garden-service/test/unit/src/commands/create/create-project.ts +++ b/garden-service/test/unit/src/commands/create/create-project.ts @@ -8,7 +8,7 @@ import { expect } from "chai" import { withDefaultGlobalOpts, TempDirectory, makeTempDir, expectError } from "../../../../helpers" -import { CreateProjectCommand } from "../../../../../src/commands/create/create-project" +import { CreateProjectCommand, defaultProjectConfigFilename } from "../../../../../src/commands/create/create-project" import { makeDummyGarden } from "../../../../../src/cli/cli" import { Garden } from "../../../../../src/garden" import { basename, join } from "path" @@ -38,13 +38,18 @@ describe("CreateProjectCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir: tmp.path, interactive: false, name: undefined }), + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: undefined, + filename: defaultProjectConfigFilename, + }), }) const { name, configPath, ignoreFileCreated, ignoreFilePath } = result! expect(name).to.equal(basename(tmp.path)) expect(ignoreFileCreated).to.be.true - expect(configPath).to.equal(join(tmp.path, "garden.yml")) + expect(configPath).to.equal(join(tmp.path, "project.garden.yml")) expect(ignoreFilePath).to.equal(join(tmp.path, ".gardenignore")) expect(await pathExists(configPath)).to.be.true expect(await pathExists(ignoreFilePath)).to.be.true @@ -71,7 +76,12 @@ describe("CreateProjectCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir: tmp.path, interactive: false, name: undefined }), + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: undefined, + filename: defaultProjectConfigFilename, + }), }) const { ignoreFileCreated, ignoreFilePath } = result! @@ -90,7 +100,12 @@ describe("CreateProjectCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir: tmp.path, interactive: false, name: undefined }), + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: undefined, + filename: defaultProjectConfigFilename, + }), }) const { ignoreFileCreated, ignoreFilePath } = result! @@ -106,7 +121,12 @@ describe("CreateProjectCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir: tmp.path, interactive: false, name: "foo" }), + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: "foo", + filename: defaultProjectConfigFilename, + }), }) const { name, configPath } = result! @@ -136,7 +156,12 @@ describe("CreateProjectCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir: tmp.path, interactive: false, name: undefined }), + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: undefined, + filename: "garden.yml", + }), }) const { name, configPath } = result! @@ -152,12 +177,32 @@ describe("CreateProjectCommand", () => { ]) }) + it("should allow overriding the default generated filename", async () => { + const { result } = await command.action({ + garden, + footerLog: garden.log, + headerLog: garden.log, + log: garden.log, + args: {}, + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: undefined, + filename: "custom.garden.yml", + }), + }) + const { configPath } = result! + + expect(configPath).to.equal(join(tmp.path, "custom.garden.yml")) + expect(await pathExists(configPath)).to.be.true + }) + it("should throw if a project is already in the directory", async () => { const existing = { kind: "Project", name: "foo", } - const configPath = join(tmp.path, "garden.yml") + const configPath = join(tmp.path, defaultProjectConfigFilename) await writeFile(configPath, safeDumpYaml(existing)) await expectError( @@ -168,7 +213,12 @@ describe("CreateProjectCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir: tmp.path, interactive: false, name: undefined }), + opts: withDefaultGlobalOpts({ + dir: tmp.path, + interactive: false, + name: undefined, + filename: defaultProjectConfigFilename, + }), }), (err) => expect(err.message).to.equal("A Garden project already exists in " + configPath) ) @@ -184,7 +234,12 @@ describe("CreateProjectCommand", () => { headerLog: garden.log, log: garden.log, args: {}, - opts: withDefaultGlobalOpts({ dir, interactive: false, name: undefined }), + opts: withDefaultGlobalOpts({ + dir, + interactive: false, + name: undefined, + filename: defaultProjectConfigFilename, + }), }), (err) => expect(err.message).to.equal(`Path ${dir} does not exist`) ) diff --git a/garden-service/test/unit/src/commands/get/get-debug-info.ts b/garden-service/test/unit/src/commands/get/get-debug-info.ts index 0d42e4dabc..26772df918 100644 --- a/garden-service/test/unit/src/commands/get/get-debug-info.ts +++ b/garden-service/test/unit/src/commands/get/get-debug-info.ts @@ -21,10 +21,13 @@ import { } from "../../../../../src/commands/get/get-debug-info" import { readdir, remove, pathExists, readJSON, readFile } from "fs-extra" import { ERROR_LOG_FILENAME } from "../../../../../src/constants" -import { join, relative } from "path" +import { join, relative, basename } from "path" import { Garden } from "../../../../../src/garden" import { LogEntry } from "../../../../../src/logger/log-entry" -import { getConfigFilePath } from "../../../../../src/util/fs" +import { defaultConfigFilename } from "../../../../../src/util/fs" +import { makeTestGarden } from "../../../../helpers" +import { getDataDir } from "../../../../helpers" +import { defaultProjectConfigFilename } from "../../../../../src/commands/create/create-project" const debugZipFileRegex = new RegExp(/debug-info-.*?.zip/) @@ -96,7 +99,7 @@ describe("GetDebugInfoCommand", () => { await collectBasicDebugInfo(garden.projectRoot, garden.gardenDirPath, log) // we first check if the main garden.yml exists - expect(await pathExists(await getConfigFilePath(gardenDebugTmp))).to.equal(true) + expect(await pathExists(join(gardenDebugTmp, defaultConfigFilename))).to.equal(true) const graph = await garden.getConfigGraph(garden.log) // Check that each module config files have been copied over and @@ -108,7 +111,7 @@ describe("GetDebugInfoCommand", () => { expect(await pathExists(join(gardenDebugTmp, moduleRelativePath))).to.equal(true) // Checks config file is copied over - expect(await pathExists(await getConfigFilePath(join(gardenDebugTmp, moduleRelativePath)))).to.equal(true) + expect(await pathExists(join(gardenDebugTmp, moduleRelativePath, defaultConfigFilename))).to.equal(true) // Checks error logs are copied over if they exist if (await pathExists(join(module.path, ERROR_LOG_FILENAME))) { @@ -116,6 +119,39 @@ describe("GetDebugInfoCommand", () => { } } }) + + it("should correctly handle custom-named config files", async () => { + const _garden = await makeTestGarden(getDataDir("test-projects", "custom-config-names")) + const debugTmp = join(_garden.gardenDirPath, TEMP_DEBUG_ROOT) + + try { + await collectBasicDebugInfo(_garden.projectRoot, _garden.gardenDirPath, log) + + // we first check if the project.garden.yml exists + expect(await pathExists(join(debugTmp, defaultProjectConfigFilename))).to.equal(true) + const graph = await _garden.getConfigGraph(_garden.log) + + // Check that each module config files have been copied over and + // the folder structure is maintained + for (const module of graph.getModules()) { + const moduleRelativePath = relative(_garden.projectRoot, module.path) + const configFilename = basename(module.configPath!) + + // Checks folder structure is maintained + expect(await pathExists(join(debugTmp, moduleRelativePath))).to.equal(true) + + // Checks config file is copied over + expect(await pathExists(join(debugTmp, moduleRelativePath, configFilename))).to.equal(true) + + // Checks error logs are copied over if they exist + if (await pathExists(join(module.path, ERROR_LOG_FILENAME))) { + expect(await pathExists(join(debugTmp, moduleRelativePath, ERROR_LOG_FILENAME))).to.equal(true) + } + } + } finally { + await cleanupTmpDebugFiles(_garden.projectRoot, _garden.gardenDirPath) + } + }) }) describe("collectSystemDiagnostic", () => { diff --git a/garden-service/test/unit/src/commands/migrate.ts b/garden-service/test/unit/src/commands/migrate.ts index 642b322d45..959c98c0c8 100644 --- a/garden-service/test/unit/src/commands/migrate.ts +++ b/garden-service/test/unit/src/commands/migrate.ts @@ -10,13 +10,13 @@ import { expect } from "chai" import { join } from "path" import tmp from "tmp-promise" import { dedent } from "../../../../src/util/string" -import cpy = require("cpy") import { sortBy } from "lodash" import { expectError, withDefaultGlobalOpts, dataDir, makeTestGardenA } from "../../../helpers" import { MigrateCommand, MigrateCommandResult, dumpSpec } from "../../../../src/commands/migrate" import { LogEntry } from "../../../../src/logger/log-entry" import { Garden } from "../../../../src/garden" import execa from "execa" +import { writeFile } from "fs-extra" describe("commands", () => { describe("migrate", () => { @@ -308,7 +308,23 @@ describe("commands", () => { }) it("should abort write if config file is dirty", async () => { await execa("git", ["init"], { cwd: tmpDir.path }) - await cpy(join(projectPath, "garden.yml"), tmpDir.path) + await writeFile( + join(tmpDir.path, "garden.yml"), + dedent` + kind: Project + name: test-project-v10-config-local-openfaas + environments: + - name: local + providers: + - name: openfaas + --- + kind: Module + name: module-a + type: local-openfaas + build: + command: [echo, project] + ` + ) await expectError( () => diff --git a/garden-service/test/unit/src/config/base.ts b/garden-service/test/unit/src/config/base.ts index 8afa8cd9d0..a24d0f83f5 100644 --- a/garden-service/test/unit/src/config/base.ts +++ b/garden-service/test/unit/src/config/base.ts @@ -7,8 +7,8 @@ */ import { expect } from "chai" -import { loadConfig, findProjectConfig } from "../../../../src/config/base" -import { resolve } from "path" +import { loadConfigResources, findProjectConfig } from "../../../../src/config/base" +import { resolve, join } from "path" import { dataDir, expectError, getDataDir } from "../../../helpers" import { DEFAULT_API_VERSION } from "../../../../src/constants" import stripAnsi = require("strip-ansi") @@ -21,17 +21,11 @@ const modulePathAMultiple = resolve(projectPathMultipleModules, "module-a") const projectPathDuplicateProjects = resolve(dataDir, "test-project-duplicate-project-config") -describe("loadConfig", () => { - it("should not throw an error if no file was found", async () => { - const parsed = await loadConfig(projectPathA, resolve(projectPathA, "non-existent-module")) - - expect(parsed).to.eql([]) - }) - +describe("loadConfigResources", () => { it("should throw a config error if the file couldn't be parsed", async () => { const projectPath = resolve(dataDir, "test-project-invalid-config") await expectError( - async () => await loadConfig(projectPath, resolve(projectPath, "invalid-syntax-module")), + async () => await loadConfigResources(projectPath, resolve(projectPath, "invalid-syntax-module", "garden.yml")), (err) => { expect(err.message).to.match(/Could not parse/) expect(err.message).to.match(/duplicated mapping key/) // include syntax erorrs in the output @@ -42,7 +36,7 @@ describe("loadConfig", () => { it("should throw if a config doesn't specify a kind", async () => { const projectPath = resolve(dataDir, "test-project-invalid-config") await expectError( - async () => await loadConfig(projectPath, resolve(projectPath, "missing-kind")), + async () => await loadConfigResources(projectPath, resolve(projectPath, "missing-kind", "garden.yml")), (err) => { expect(err.message).to.equal("Missing `kind` field in config at missing-kind/garden.yml") } @@ -52,7 +46,7 @@ describe("loadConfig", () => { it("should throw if a config specifies an invalid kind", async () => { const projectPath = resolve(dataDir, "test-project-invalid-config") await expectError( - async () => await loadConfig(projectPath, resolve(projectPath, "invalid-config-kind")), + async () => await loadConfigResources(projectPath, resolve(projectPath, "invalid-config-kind", "garden.yml")), (err) => { expect(err.message).to.equal("Unknown config kind banana in invalid-config-kind/garden.yml") } @@ -62,7 +56,7 @@ describe("loadConfig", () => { it("should throw if a module config doesn't specify a type", async () => { const projectPath = resolve(dataDir, "test-project-invalid-config") await expectError( - async () => await loadConfig(projectPath, resolve(projectPath, "missing-type")), + async () => await loadConfigResources(projectPath, resolve(projectPath, "missing-type", "garden.yml")), (err) => { expect(stripAnsi(err.message)).to.equal( "Error validating module (missing-type/garden.yml): key .type is required" @@ -74,7 +68,7 @@ describe("loadConfig", () => { it("should throw if a module config doesn't specify a name", async () => { const projectPath = resolve(dataDir, "test-project-invalid-config") await expectError( - async () => await loadConfig(projectPath, resolve(projectPath, "missing-name")), + async () => await loadConfigResources(projectPath, resolve(projectPath, "missing-name", "garden.yml")), (err) => { expect(stripAnsi(err.message)).to.equal( "Error validating module (missing-name/garden.yml): key .name is required" @@ -85,8 +79,8 @@ describe("loadConfig", () => { // TODO: test more cases it("should load and parse a project config", async () => { - const parsed = await loadConfig(projectPathA, projectPathA) const configPath = resolve(projectPathA, "garden.yml") + const parsed = await loadConfigResources(projectPathA, configPath) expect(parsed).to.eql([ { @@ -120,8 +114,8 @@ describe("loadConfig", () => { }) it("should load and parse a module config", async () => { - const parsed = await loadConfig(projectPathA, modulePathA) const configPath = resolve(modulePathA, "garden.yml") + const parsed = await loadConfigResources(projectPathA, configPath) expect(parsed).to.eql([ { @@ -173,8 +167,8 @@ describe("loadConfig", () => { }) it("should load and parse a config file defining a project and a module", async () => { - const parsed = await loadConfig(projectPathMultipleModules, projectPathMultipleModules) const configPath = resolve(projectPathMultipleModules, "garden.yml") + const parsed = await loadConfigResources(projectPathMultipleModules, configPath) expect(parsed).to.eql([ { @@ -226,8 +220,8 @@ describe("loadConfig", () => { }) it("should load and parse a config file defining multiple modules", async () => { - const parsed = await loadConfig(projectPathMultipleModules, modulePathAMultiple) const configPath = resolve(modulePathAMultiple, "garden.yml") + const parsed = await loadConfigResources(projectPathMultipleModules, configPath) expect(parsed).to.eql([ { @@ -293,14 +287,15 @@ describe("loadConfig", () => { it("should load a project config with a top-level provider field", async () => { const projectPath = getDataDir("test-projects", "new-provider-spec") - const parsed = await loadConfig(projectPath, projectPath) + const configPath = resolve(projectPath, "garden.yml") + const parsed = await loadConfigResources(projectPath, configPath) expect(parsed).to.eql([ { apiVersion: "garden.io/v0", kind: "Project", path: projectPath, - configPath: resolve(projectPath, "garden.yml"), + configPath, name: "test-project-a", environments: [{ name: "local" }, { name: "other" }], providers: [{ name: "test-plugin", environments: ["local"] }, { name: "test-plugin-b" }], @@ -308,36 +303,35 @@ describe("loadConfig", () => { ]) }) - it("should throw an error when parsing a config file defining multiple projects", async () => { + it("should throw if config file is not found", async () => { await expectError( - async () => await loadConfig(projectPathDuplicateProjects, projectPathDuplicateProjects), + async () => await loadConfigResources("/thisdoesnotexist", "/thisdoesnotexist"), (err) => { - expect(err.message).to.match(/Multiple project declarations/) + expect(err.message).to.equal("Could not find configuration file at /thisdoesnotexist") } ) }) - it("should return [] if config file is not found", async () => { - const parsed = await loadConfig("/thisdoesnotexist", "/thisdoesnotexist") - expect(parsed).to.eql([]) - }) - it("should ignore empty documents in multi-doc YAML", async () => { const path = resolve(dataDir, "test-projects", "empty-doc") - const parsed = await loadConfig(path, path) + const configPath = resolve(path, "garden.yml") + const parsed = await loadConfigResources(path, configPath) + expect(parsed).to.eql([ { apiVersion: DEFAULT_API_VERSION, kind: "Project", name: "foo", path, - configPath: resolve(path, "garden.yml"), + configPath, }, ]) }) }) describe("findProjectConfig", async () => { + const customConfigPath = getDataDir("test-projects", "custom-config-names") + it("should find the project config when path is projectRoot", async () => { const project = await findProjectConfig(projectPathA) expect(project && project.path).to.eq(projectPathA) @@ -348,4 +342,24 @@ describe("findProjectConfig", async () => { const project = await findProjectConfig(modulePathA) expect(project && project.path).to.eq(projectPathA) }) + + it("should find the project config when path is projectRoot and config is in a custom-named file", async () => { + const project = await findProjectConfig(customConfigPath) + expect(project && project.path).to.eq(customConfigPath) + }) + + it("should find the project root from a subdir of projectRoot and config is in a custom-named file", async () => { + const modulePath = join(customConfigPath, "module-a") + const project = await findProjectConfig(modulePath) + expect(project && project.path).to.eq(customConfigPath) + }) + + it("should throw an error if multiple projects are found", async () => { + await expectError( + async () => await findProjectConfig(projectPathDuplicateProjects), + (err) => { + expect(err.message).to.match(/Multiple project declarations found/) + } + ) + }) }) diff --git a/garden-service/test/unit/src/garden.ts b/garden-service/test/unit/src/garden.ts index 3c7c2e3d8f..00e346e94f 100644 --- a/garden-service/test/unit/src/garden.ts +++ b/garden-service/test/unit/src/garden.ts @@ -101,11 +101,6 @@ describe("Garden", () => { expect(gardenCustomDir.moduleExcludePatterns).to.include("custom/garden-dir/**/*") }) - it("should throw if a project has config files with yaml and yml extensions in the same dir", async () => { - const path = getDataDir("test-project-duplicate-yaml-file-extensions") - await expectError(async () => makeTestGarden(path), "validation") - }) - it("should parse and resolve the config from the project root", async () => { const garden = await makeTestGardenA() const projectRoot = garden.projectRoot @@ -146,6 +141,12 @@ describe("Garden", () => { }) }) + it("should load a project config in a custom-named config file", async () => { + const projectRoot = getDataDir("test-projects", "custom-config-names") + const garden = await makeTestGarden(projectRoot) + expect(garden.projectRoot).to.equal(projectRoot) + }) + it("should resolve templated env variables in project config", async () => { process.env.TEST_PROVIDER_TYPE = "test-plugin" process.env.TEST_VARIABLE = "banana" @@ -2144,9 +2145,26 @@ describe("Garden", () => { ]) }) + it("should scan and add modules contained in custom-named config files", async () => { + const garden = await makeTestGarden(resolve(dataDir, "test-projects", "custom-config-names")) + await garden.scanAndAddConfigs() + + const modules = await garden.resolveModules({ log: garden.log }) + expect(getNames(modules).sort()).to.eql(["module-a", "module-b"]) + }) + + it("should scan and add workflows contained in custom-named config files", async () => { + const garden = await makeTestGarden(resolve(dataDir, "test-projects", "custom-config-names")) + await garden.scanAndAddConfigs() + + const workflows = garden.getWorkflowConfigs() + expect(getNames(workflows)).to.eql(["workflow-a", "workflow-b"]) + }) + it("should scan and add modules for projects with external project sources", async () => { const garden = await makeExtProjectSourcesGarden() await garden.scanAndAddConfigs() + const modules = await garden.resolveModules({ log: garden.log }) expect(getNames(modules).sort()).to.eql(["module-a", "module-b", "module-c"]) }) @@ -2163,12 +2181,6 @@ describe("Garden", () => { ) }) - it("should scan and add modules with config files with yaml and yml extensions", async () => { - const garden = await makeTestGarden(getDataDir("test-project-yaml-file-extensions")) - const modules = await garden.resolveModules({ log: garden.log }) - expect(getNames(modules).sort()).to.eql(["module-yaml", "module-yml"]) - }) - it("should respect the modules.include and modules.exclude fields, if specified", async () => { const projectRoot = getDataDir("test-projects", "project-include-exclude") const garden = await makeTestGarden(projectRoot) @@ -2238,23 +2250,6 @@ describe("Garden", () => { }) }) - describe("loadConfigs", () => { - it("should resolve module by absolute path", async () => { - const garden = await makeTestGardenA() - const path = join(projectRootA, "module-a") - - const module = (await (garden).loadConfigs(path))[0] - expect(module!.name).to.equal("module-a") - }) - - it("should resolve module by relative path to project root", async () => { - const garden = await makeTestGardenA() - - const module = (await (garden).loadConfigs("./module-a"))[0] - expect(module!.name).to.equal("module-a") - }) - }) - describe("resolveModules", () => { it("should throw if a module references itself in a template string", async () => { const projectRoot = resolve(dataDir, "test-projects", "module-self-ref") diff --git a/garden-service/test/unit/src/util/fs.ts b/garden-service/test/unit/src/util/fs.ts index 5a30802ef1..0d5b098d23 100644 --- a/garden-service/test/unit/src/util/fs.ts +++ b/garden-service/test/unit/src/util/fs.ts @@ -7,14 +7,13 @@ */ import { expect } from "chai" -import { join, basename } from "path" -import { getDataDir, expectError, makeTestGardenA } from "../../../helpers" +import { join } from "path" +import { getDataDir, makeTestGardenA, makeTestGarden } from "../../../helpers" import { scanDirectory, toCygwinPath, getChildDirNames, isConfigFilename, - getConfigFilePath, getWorkingCopyId, findConfigPathsInPath, detectModuleOverlap, @@ -22,9 +21,6 @@ import { import { withDir } from "tmp-promise" import { ModuleConfig } from "../../../../src/config/module" -const projectYamlFileExtensions = getDataDir("test-project-yaml-file-extensions") -const projectDuplicateYamlFileExtensions = getDataDir("test-project-duplicate-yaml-file-extensions") - describe("util", () => { describe("detectModuleOverlap", () => { it("should detect if modules have the same root", () => { @@ -237,35 +233,12 @@ describe("util", () => { }) }) - describe("getConfigFilePath", () => { - context("name of the file is garden.yml", () => { - it("should return the full path to the config file", async () => { - const testPath = join(projectYamlFileExtensions, "module-yml") - expect(await getConfigFilePath(testPath)).to.eql(join(testPath, "garden.yml")) - }) - }) - context("name of the file is garden.yaml", () => { - it("should return the full path to the config file", async () => { - const testPath = join(projectYamlFileExtensions, "module-yml") - expect(await getConfigFilePath(testPath)).to.eql(join(testPath, "garden.yml")) - }) - }) - it("should throw if multiple valid config files found at the given path", async () => { - await expectError(() => getConfigFilePath(projectDuplicateYamlFileExtensions), "validation") - }) - it("should return a valid default path if no config file found at the given path", async () => { - const testPath = join(projectYamlFileExtensions, "module-no-config") - const result = await getConfigFilePath(testPath) - expect(isConfigFilename(basename(result))).to.be.true - }) - }) - describe("isConfigFilename", () => { it("should return true if the name of the file is garden.yaml", async () => { - expect(await isConfigFilename("garden.yaml")).to.be.true + expect(isConfigFilename("garden.yaml")).to.be.true }) it("should return true if the name of the file is garden.yml", async () => { - expect(await isConfigFilename("garden.yml")).to.be.true + expect(isConfigFilename("garden.yml")).to.be.true }) it("should return false otherwise", async () => { const badNames = ["agarden.yml", "garden.ymla", "garden.yaaml", "garden.ml"] @@ -300,7 +273,7 @@ describe("util", () => { }) describe("findConfigPathsInPath", () => { - it("should find all garden configs in a directory", async () => { + it("should recursively find all garden configs in a directory", async () => { const garden = await makeTestGardenA() const files = await findConfigPathsInPath({ vcs: garden.vcs, @@ -315,6 +288,21 @@ describe("util", () => { ]) }) + it("should find custom-named garden configs", async () => { + const garden = await makeTestGarden(getDataDir("test-projects", "custom-config-names")) + const files = await findConfigPathsInPath({ + vcs: garden.vcs, + dir: garden.projectRoot, + log: garden.log, + }) + expect(files).to.eql([ + join(garden.projectRoot, "module-a", "garden.yml"), + join(garden.projectRoot, "module-b", "module-b.garden.yaml"), + join(garden.projectRoot, "project.garden.yml"), + join(garden.projectRoot, "workflows.garden.yml"), + ]) + }) + it("should respect the include option, if specified", async () => { const garden = await makeTestGardenA() const include = ["module-a/**/*"] diff --git a/garden-service/test/unit/src/watch.ts b/garden-service/test/unit/src/watch.ts index 4cad2889c1..7e8c819f5a 100644 --- a/garden-service/test/unit/src/watch.ts +++ b/garden-service/test/unit/src/watch.ts @@ -21,7 +21,6 @@ import { } from "../../helpers" import { CacheContext, pathToCacheContext } from "../../../src/cache" import { remove, pathExists, writeFile } from "fs-extra" -import { getConfigFilePath } from "../../../src/util/fs" import { LinkModuleCommand } from "../../../src/commands/link/module" import { LinkSourceCommand } from "../../../src/commands/link/source" import { sleep } from "../../../src/util/util" @@ -80,14 +79,18 @@ describe("Watcher", () => { return garden.events.eventLog.filter((e) => !e.name.startsWith("task")) } + function getConfigFilePath(path: string) { + return join(path, "garden.yml") + } + it("should emit a moduleConfigChanged changed event when module config is changed", async () => { - const path = await getConfigFilePath(modulePath) + const path = getConfigFilePath(modulePath) emitEvent(garden, "change", path) expect(getEventLog()).to.eql([{ name: "moduleConfigChanged", payload: { names: ["module-a"], path } }]) }) it("should emit a moduleConfigChanged event when module config is changed and include field is set", async () => { - const path = await getConfigFilePath(includeModulePath) + const path = getConfigFilePath(includeModulePath) emitEvent(garden, "change", path) expect(getEventLog()).to.eql([ { @@ -98,44 +101,44 @@ describe("Watcher", () => { }) it("should clear all module caches when a module config is changed", async () => { - const path = await getConfigFilePath(modulePath) + const path = getConfigFilePath(modulePath) emitEvent(garden, "change", path) expect(garden.cache.getByContext(moduleContext)).to.eql(new Map()) }) it("should emit a projectConfigChanged changed event when project config is changed", async () => { - const path = await getConfigFilePath(garden.projectRoot) + const path = getConfigFilePath(garden.projectRoot) emitEvent(garden, "change", path) expect(getEventLog()).to.eql([{ name: "projectConfigChanged", payload: {} }]) }) it("should emit a projectConfigChanged changed event when project config is removed", async () => { - const path = await getConfigFilePath(garden.projectRoot) + const path = getConfigFilePath(garden.projectRoot) emitEvent(garden, "unlink", path) await waitForEvent("projectConfigChanged") expect(getEventLog()).to.eql([{ name: "projectConfigChanged", payload: {} }]) }) it("should emit a projectConfigChanged changed event when ignore files are changed", async () => { - const path = join(await getConfigFilePath(garden.projectRoot), ".gardenignore") + const path = join(getConfigFilePath(garden.projectRoot), ".gardenignore") emitEvent(garden, "change", path) expect(getEventLog()).to.eql([{ name: "projectConfigChanged", payload: {} }]) }) it("should clear all module caches when project config is changed", async () => { - const path = await getConfigFilePath(garden.projectRoot) + const path = getConfigFilePath(garden.projectRoot) emitEvent(garden, "change", path) expect(garden.cache.getByContext(moduleContext)).to.eql(new Map()) }) it("should emit a configAdded event when adding a garden.yml file", async () => { - const path = await getConfigFilePath(join(garden.projectRoot, "module-b")) + const path = getConfigFilePath(join(garden.projectRoot, "module-b")) emitEvent(garden, "add", path) expect(await waitForEvent("configAdded")).to.eql({ path }) }) it("should emit a configRemoved event when removing a garden.yml file", async () => { - const path = await getConfigFilePath(join(garden.projectRoot, "module-a")) + const path = getConfigFilePath(join(garden.projectRoot, "module-a")) emitEvent(garden, "unlink", path) await waitForEvent("configRemoved") expect(getEventLog()).to.eql([{ name: "configRemoved", payload: { path } }]) @@ -223,7 +226,7 @@ describe("Watcher", () => { it("should emit a configAdded event when a directory is added that contains a garden.yml file", async () => { emitEvent(garden, "addDir", modulePath) expect(await waitForEvent("configAdded")).to.eql({ - path: await getConfigFilePath(modulePath), + path: getConfigFilePath(modulePath), }) })