diff --git a/docs/guides/configuration-files.md b/docs/guides/configuration-files.md index a86ef955f4..3c05594bb4 100644 --- a/docs/guides/configuration-files.md +++ b/docs/guides/configuration-files.md @@ -353,6 +353,8 @@ Here we only scan the `modules` directory, but exclude the `modules/tmp` directo If you specify a list with `include`, only those patterns are included. If you then specify one or more `exclude` patterns, those are filtered out of the ones matched by `include`. If you _only_ specify `exclude`, those patterns will be filtered out of all paths in the project directory. +The `modules.exclude` field is also used to limit the number of files and directories Garden watches for changes while running. Use that if you have a large number of files/directories in your project that you do not need to watch, or if you are seeing excessive CPU/RAM usage. The `modules.include` field has no effect on which paths Garden watches for changes. + #### Module include/exclude By default, all files in the same directory as a module configuration file are included as source files for that module. Sometimes you need more granular control over the context, not least if you have multiple modules in the same directory. @@ -375,6 +377,8 @@ Here we only include the `Dockerfile` and all the `.py` files under `my-sources/ If you specify a list with `include`, only those files/patterns are included. If you then specify one or more `exclude` files or patterns, those are filtered out of the files matched by `include`. If you _only_ specify `exclude`, those patterns will be filtered out of all files in the module directory. +Note that the module `include` and `exclude` fields have no effect on which paths Garden watches for changes. Use the project `modules.exclude` field (described above) for that purpose. + #### .ignore files By default, Garden respects `.gitignore` and `.gardenignore` files and excludes any patterns matched in those files. diff --git a/docs/reference/config.md b/docs/reference/config.md index 3168b4a92a..a6eb426e81 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -178,6 +178,10 @@ Note that you can also _exclude_ path using the `exclude` field or by placing `. source tree, which use the same format as `.gitignore` files. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories) for details. +Unlike the `exclude` field, the paths/globs specified here have _no effect_ on which files and directories +Garden watches for changes. Use the `exclude` field to affect those, if you have large directories that +should not be watched for changes. + Also note that specifying an empty list here means _no paths_ should be included. | Type | Required | @@ -198,7 +202,26 @@ modules: [modules](#modules) > exclude Specify a list of POSIX-style paths or glob patterns that should be excluded when scanning for modules. -Note that you can also explicitly _include_ files using the `include` field. If you also specify the `include` field, the paths/patterns specified here are filtered from the files matched by `include`. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories) for details. + +The filters here also affect which files and directories are watched for changes. So if you have a large number +of directories in your project that should not be watched, you should specify them here. + +For example, you might want to exclude large vendor directories in your project from being scanned and +watched: + +```yaml +modules: + exclude: + - node_modules/**/* + - vendor/**/* +``` + +Note that you can also explicitly _include_ files using the `include` field. If you also specify the +`include` field, the paths/patterns specified here are filtered from the files matched by `include`. + +The `include` field does _not_ affect which files are watched. + +See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories) for details. | Type | Required | | --------------- | -------- | @@ -527,6 +550,10 @@ Note that you can also explicitly _include_ files using the `include` field. If `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details. +Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files +and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have +large directories that should not be watched for changes. + | Type | Required | | --------------- | -------- | | `array[string]` | No | diff --git a/docs/reference/module-types/container.md b/docs/reference/module-types/container.md index dcf7f6d98a..4b80b34917 100644 --- a/docs/reference/module-types/container.md +++ b/docs/reference/module-types/container.md @@ -103,6 +103,10 @@ Note that you can also explicitly _include_ files using the `include` field. If `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details. +Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files +and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have +large directories that should not be watched for changes. + | Type | Required | | --------------- | -------- | | `array[string]` | No | diff --git a/docs/reference/module-types/exec.md b/docs/reference/module-types/exec.md index 753740fd1c..09a1e43271 100644 --- a/docs/reference/module-types/exec.md +++ b/docs/reference/module-types/exec.md @@ -106,6 +106,10 @@ Note that you can also explicitly _include_ files using the `include` field. If `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details. +Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files +and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have +large directories that should not be watched for changes. + | Type | Required | | --------------- | -------- | | `array[string]` | No | diff --git a/docs/reference/module-types/helm.md b/docs/reference/module-types/helm.md index 64ccebde1b..3d86499be8 100644 --- a/docs/reference/module-types/helm.md +++ b/docs/reference/module-types/helm.md @@ -98,6 +98,10 @@ Note that you can also explicitly _include_ files using the `include` field. If `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details. +Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files +and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have +large directories that should not be watched for changes. + | Type | Required | | --------------- | -------- | | `array[string]` | No | diff --git a/docs/reference/module-types/kubernetes.md b/docs/reference/module-types/kubernetes.md index eb6a845a5f..b6f7518df7 100644 --- a/docs/reference/module-types/kubernetes.md +++ b/docs/reference/module-types/kubernetes.md @@ -106,6 +106,10 @@ Note that you can also explicitly _include_ files using the `include` field. If `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details. +Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files +and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have +large directories that should not be watched for changes. + | Type | Required | | --------------- | -------- | | `array[string]` | No | diff --git a/docs/reference/module-types/maven-container.md b/docs/reference/module-types/maven-container.md index af11d4aa0a..466c137560 100644 --- a/docs/reference/module-types/maven-container.md +++ b/docs/reference/module-types/maven-container.md @@ -108,6 +108,10 @@ Note that you can also explicitly _include_ files using the `include` field. If `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details. +Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files +and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have +large directories that should not be watched for changes. + | Type | Required | | --------------- | -------- | | `array[string]` | No | diff --git a/docs/reference/module-types/openfaas.md b/docs/reference/module-types/openfaas.md index 24527e7244..647e564218 100644 --- a/docs/reference/module-types/openfaas.md +++ b/docs/reference/module-types/openfaas.md @@ -98,6 +98,10 @@ Note that you can also explicitly _include_ files using the `include` field. If `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details. +Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files +and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have +large directories that should not be watched for changes. + | Type | Required | | --------------- | -------- | | `array[string]` | No | diff --git a/docs/reference/module-types/terraform.md b/docs/reference/module-types/terraform.md index 90bcf2bd33..d83f78090d 100644 --- a/docs/reference/module-types/terraform.md +++ b/docs/reference/module-types/terraform.md @@ -112,6 +112,10 @@ Note that you can also explicitly _include_ files using the `include` field. If `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details. +Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files +and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have +large directories that should not be watched for changes. + | Type | Required | | --------------- | -------- | | `array[string]` | No | diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 3a01356309..b29030bba0 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -7,6 +7,12 @@ title: Troubleshooting _This section could (obviously) use more work. Contributions are most appreciated!_ +## I have a huge number of files in my repository and Garden is eating all my CPU/RAM + +This issue often comes up on Linux, and in other scenarios where the filesystem doesn't support event-based file watching. + +Thankfully, you can in most cases avoid this problem using the `modules.exclude` field in your project config, and/or the `exclude` field in your individual module configs. See the [Including/excluding files and directories](./using-garden/configuration-files#including-excluding-files-and-directories) section in our Configuration Files guide for details. + ## I'm getting an "EPERM: operation not permitted, rename..." error on Windows This is a known issue with Windows and may affect many Node.js applications (and possibly others). diff --git a/garden-service/package-lock.json b/garden-service/package-lock.json index 09de70c3b0..7eb441ba87 100644 --- a/garden-service/package-lock.json +++ b/garden-service/package-lock.json @@ -1286,9 +1286,9 @@ "integrity": "sha1-q8av7tzqUugJzcA3au0845Y10X8=" }, "anymatch": { - "version": "3.0.3", - "resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.0.3.tgz", - "integrity": "sha512-c6IvoeBECQlMVuYUjSwimnhmztImpErfxJzWZhIQinIvQWoGOnB0dLIgifbPHQt5heS6mNlaZG16f06H3C8t1g==", + "version": "3.1.1", + "resolved": "https://registry.npmjs.org/anymatch/-/anymatch-3.1.1.tgz", + "integrity": "sha512-mM8522psRCqzV+6LhomX5wgp25YVibjh8Wj23I5RPkPppSVSjyKD2A2mBJmWGa+KN7f2D6LNh9jkBCeyLktzjg==", "requires": { "normalize-path": "^3.0.0", "picomatch": "^2.0.4" @@ -2260,18 +2260,18 @@ } }, "chokidar": { - "version": "3.0.2", - "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.0.2.tgz", - "integrity": "sha512-c4PR2egjNjI1um6bamCQ6bUNPDiyofNQruHvKgHQ4gDUP/ITSVSzNsiI5OWtHOsX323i5ha/kk4YmOZ1Ktg7KA==", - "requires": { - "anymatch": "^3.0.1", - "braces": "^3.0.2", - "fsevents": "^2.0.6", - "glob-parent": "^5.0.0", - "is-binary-path": "^2.1.0", - "is-glob": "^4.0.1", - "normalize-path": "^3.0.0", - "readdirp": "^3.1.1" + "version": "3.3.0", + "resolved": "https://registry.npmjs.org/chokidar/-/chokidar-3.3.0.tgz", + "integrity": "sha512-dGmKLDdT3Gdl7fBUe8XK+gAtGmzy5Fn0XkkWQuYxGIgWVPPse2CxFA5mtrlD0TOHaHjEUqkWNyP1XdHoJES/4A==", + "requires": { + "anymatch": "~3.1.1", + "braces": "~3.0.2", + "fsevents": "~2.1.1", + "glob-parent": "~5.1.0", + "is-binary-path": "~2.1.0", + "is-glob": "~4.0.1", + "normalize-path": "~3.0.0", + "readdirp": "~3.2.0" } }, "chownr": { @@ -4367,9 +4367,9 @@ "integrity": "sha1-FQStJSMVjKpA20onh8sBQRmU6k8=" }, "fsevents": { - "version": "2.0.7", - "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.0.7.tgz", - "integrity": "sha512-a7YT0SV3RB+DjYcppwVDLtn13UQnmg0SWZS7ezZD0UjnLwXmy8Zm21GMVGLaFGimIqcvyMQaOJBrop8MyOp1kQ==", + "version": "2.1.1", + "resolved": "https://registry.npmjs.org/fsevents/-/fsevents-2.1.1.tgz", + "integrity": "sha512-4FRPXWETxtigtJW/gxzEDsX1LVbPAM93VleB83kZB+ellqbHMkyt2aJfuzNLRvFPnGi6bcE5SvfxgbXPeKteJw==", "optional": true }, "fstream": { @@ -4513,9 +4513,9 @@ } }, "glob-parent": { - "version": "5.0.0", - "resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.0.0.tgz", - "integrity": "sha512-Z2RwiujPRGluePM6j699ktJYxmPpJKCfpGA13jz2hmFZC7gKetzrWvg5KN3+OsIFmydGyZ1AVwERCq1w/ZZwRg==", + "version": "5.1.0", + "resolved": "https://registry.npmjs.org/glob-parent/-/glob-parent-5.1.0.tgz", + "integrity": "sha512-qjtRgnIVmOfnKUE3NJAQEdk+lKrxfw8t5ke7SXtfMTHcjsBfOfWXCQfdb30zfDoZQ2IRSIiidmjtbHZPZ++Ihw==", "requires": { "is-glob": "^4.0.1" } @@ -8701,9 +8701,9 @@ "integrity": "sha1-Ywn04OX6kT7BxpMHrjZLSzd8nns=" }, "picomatch": { - "version": "2.0.7", - "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.0.7.tgz", - "integrity": "sha512-oLHIdio3tZ0qH76NybpeneBhYVj0QFTfXEFTc/B3zKQspYfYYkWYgFsmzo+4kvId/bQRcNkVeguI3y+CD22BtA==" + "version": "2.1.0", + "resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.1.0.tgz", + "integrity": "sha512-uhnEDzAbrcJ8R3g2fANnSuXZMBtkpSjxTTgn2LeSiQlfmq72enQJWdQllXW24MBLYnA1SBD2vfvx2o0Zw3Ielw==" }, "pify": { "version": "3.0.0", @@ -9143,9 +9143,9 @@ } }, "readdirp": { - "version": "3.1.2", - "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.1.2.tgz", - "integrity": "sha512-8rhl0xs2cxfVsqzreYCvs8EwBfn/DhVdqtoLmw19uI3SC5avYX9teCurlErfpPXGmYtMHReGaP2RsLnFvz/lnw==", + "version": "3.2.0", + "resolved": "https://registry.npmjs.org/readdirp/-/readdirp-3.2.0.tgz", + "integrity": "sha512-crk4Qu3pmXwgxdSgGhgA/eXiJAPQiX4GMOZZMXnqKxHX7TaoL+3gQVo/WeuAiogr07DpnfjIMpXXa+PAIvwPGQ==", "requires": { "picomatch": "^2.0.4" } diff --git a/garden-service/package.json b/garden-service/package.json index fe9e31e39c..db6790a630 100644 --- a/garden-service/package.json +++ b/garden-service/package.json @@ -36,7 +36,7 @@ "certpem": "^1.1.3", "chalk": "^2.4.2", "child-process-promise": "^2.2.1", - "chokidar": "^3.0.2", + "chokidar": "^3.3.0", "ci-info": "^2.0.0", "circular-json": "^0.5.9", "cli-cursor": "^3.1.0", @@ -218,7 +218,7 @@ "e2e": "cd test/e2e && ../../bin/garden test", "e2e-project": "node build/test/e2e/e2e-project.js", "pkg": "./bin/build-pkg.sh", - "test": "node_modules/.bin/mocha", + "test": "node_modules/.bin/mocha --opts test/mocha.opts", "tsc": "tsc -p . --outDir build --declaration --declarationMap --sourceMap", "view-report": "open coverage/index.html", "watch": "npm run tsc -- -w" @@ -230,4 +230,4 @@ ] }, "gitHead": "b0647221a4d2ff06952bae58000b104215aed922" -} +} \ No newline at end of file diff --git a/garden-service/src/config/module.ts b/garden-service/src/config/module.ts index d6dd49cf1b..38f5bcac23 100644 --- a/garden-service/src/config/module.ts +++ b/garden-service/src/config/module.ts @@ -140,7 +140,12 @@ export const baseModuleSpecSchema = joi Note that you can also explicitly _include_ files using the \`include\` field. If you also specify the \`include\` field, the files/patterns specified here are filtered from the files matched by \`include\`. See the - [Configuration Files guide](${includeGuideLink})for details.` + [Configuration Files guide](${includeGuideLink})for details. + + Unlike the \`modules.exclude\` field in the project config, the filters here have _no effect_ on which files + and directories are watched for changes. Use the project \`modules.exclude\` field to affect those, if you have + large directories that should not be watched for changes. + ` ) .example([["tmp/**/*", "*.log"], {}]), repositoryUrl: joiRepositoryUrl().description( diff --git a/garden-service/src/config/project.ts b/garden-service/src/config/project.ts index 995da91bee..816ed742e7 100644 --- a/garden-service/src/config/project.ts +++ b/garden-service/src/config/project.ts @@ -166,12 +166,17 @@ const projectModulesSchema = joi.object().keys({ .array() .items(joi.string().posixPath({ subPathOnly: true })) .description( - dedent`Specify a list of POSIX-style paths or globs that should be scanned for Garden modules. + dedent` + Specify a list of POSIX-style paths or globs that should be scanned for Garden modules. Note that you can also _exclude_ path using the \`exclude\` field or by placing \`.gardenignore\` files in your source tree, which use the same format as \`.gitignore\` files. See the [Configuration Files guide](${includeGuideLink}) for details. + Unlike the \`exclude\` field, the paths/globs specified here have _no effect_ on which files and directories + Garden watches for changes. Use the \`exclude\` field to affect those, if you have large directories that + should not be watched for changes. + Also note that specifying an empty list here means _no paths_ should be included.` ) .example([["modules/**/*"], {}]), @@ -179,11 +184,29 @@ const projectModulesSchema = joi.object().keys({ .array() .items(joi.string().posixPath({ subPathOnly: true })) .description( - deline`Specify a list of POSIX-style paths or glob patterns that should be excluded when scanning for modules. + dedent` + Specify a list of POSIX-style paths or glob patterns that should be excluded when scanning for modules. + + The filters here also affect which files and directories are watched for changes. So if you have a large number + of directories in your project that should not be watched, you should specify them here. + + For example, you might want to exclude large vendor directories in your project from being scanned and + watched: + + \`\`\`yaml + modules: + exclude: + - node_modules/**/* + - vendor/**/* + \`\`\` Note that you can also explicitly _include_ files using the \`include\` field. If you also specify the - \`include\` field, the paths/patterns specified here are filtered from the files matched by \`include\`. See the - [Configuration Files guide](${includeGuideLink}) for details.` + \`include\` field, the paths/patterns specified here are filtered from the files matched by \`include\`. + + The \`include\` field does _not_ affect which files are watched. + + See the [Configuration Files guide](${includeGuideLink}) for details. + ` ) .example([["public/**/*", "tmp/**/*"], {}]), }) diff --git a/garden-service/src/garden.ts b/garden-service/src/garden.ts index 2ac5d87354..a324071721 100644 --- a/garden-service/src/garden.ts +++ b/garden-service/src/garden.ts @@ -256,7 +256,7 @@ export class Garden { * Clean up before shutting down. */ async close() { - this.watcher && this.watcher.stop() + this.watcher && (await this.watcher.stop()) } getPluginContext(provider: Provider) { diff --git a/garden-service/src/watch.ts b/garden-service/src/watch.ts index 18580d259e..bf485d59fa 100644 --- a/garden-service/src/watch.ts +++ b/garden-service/src/watch.ts @@ -7,34 +7,20 @@ */ import { watch, FSWatcher } from "chokidar" -import { parse, basename } from "path" +import { parse, basename, resolve } from "path" import { pathToCacheContext } from "./cache" import { Module } from "./types/module" import { Garden } from "./garden" import { LogEntry } from "./logger/log-entry" -import { registerCleanupFunction, sleep } from "./util/util" +import { sleep } from "./util/util" import { some } from "lodash" import { isConfigFilename, matchPath } from "./util/fs" import Bluebird from "bluebird" import { InternalError } from "./exceptions" +import { EventEmitter } from "events" // How long we wait between processing added files and directories -const DEFAULT_BUFFER_INTERVAL = 500 - -// IMPORTANT: We must use a single global instance of the watcher, because we may otherwise get -// segmentation faults on macOS! See https://github.com/fsevents/fsevents/issues/273 -let watcher: FSWatcher | undefined - -// Export so that we can clean up the global watcher instance when running tests -export function cleanUpGlobalWatcher() { - if (watcher) { - watcher.close() - watcher = undefined - } -} - -// The process hangs after tests if we don't do this -registerCleanupFunction("stop watcher", cleanUpGlobalWatcher) +const DEFAULT_BUFFER_INTERVAL = 400 export type ChangeHandler = (module: Module | null, configChanged: boolean) => Promise @@ -50,7 +36,7 @@ interface ChangedPath { * Wrapper around the Chokidar file watcher. Emits events on `garden.events` when project files are changed. * This needs to be enabled by calling the `.start()` method, and stopped with the `.stop()` method. */ -export class Watcher { +export class Watcher extends EventEmitter { private watcher: FSWatcher private buffer: { [path: string]: ChangedPath } private running: boolean @@ -63,18 +49,20 @@ export class Watcher { private modules: Module[], private bufferInterval: number = DEFAULT_BUFFER_INTERVAL ) { + super() this.buffer = {} this.running = false this.processing = false this.start() } - stop(): void { + async stop() { this.running = false if (this.watcher) { this.log.debug(`Watcher: Clearing handlers`) this.watcher.removeAllListeners() + await this.watcher.close() delete this.watcher } } @@ -82,7 +70,9 @@ export class Watcher { start() { this.log.debug(`Watcher: Watching paths ${this.paths.join(", ")}`) - if (watcher === undefined) { + this.running = true + + if (!this.watcher) { // Make sure that fsevents works when we're on macOS. This has come up before without us noticing, which has // a dramatic performance impact, so it's best if we simply throw here so that our tests catch such issues. if (process.platform === "darwin") { @@ -95,21 +85,26 @@ export class Watcher { } } - watcher = watch(this.paths, { + // Collect all the configured excludes and pass to the watcher. + // This allows chokidar to optimize polling based on the exclusions. + // See https://github.com/garden-io/garden/issues/1269. + // TODO: see if we can extract paths from dotignore files as well (we'd have to deal with negations etc. somehow). + const projectExcludes = this.garden.moduleExcludePatterns.map((p) => resolve(this.garden.projectRoot, p)) + // TODO: filter paths based on module excludes as well + // (requires more complex logic to handle overlapping module sources). + // const moduleExcludes = flatten(this.modules.map((m) => (m.exclude || []).map((p) => resolve(m.path, p)))) + + this.watcher = watch(this.paths, { ignoreInitial: true, ignorePermissionErrors: true, persistent: true, awaitWriteFinish: { - stabilityThreshold: 500, - pollInterval: 200, + stabilityThreshold: 300, + pollInterval: 100, }, + ignored: [...projectExcludes], + useFsEvents: false, }) - } - - this.running = true - - if (!this.watcher) { - this.watcher = watcher this.watcher .on("add", this.makeFileAddedHandler()) @@ -117,6 +112,13 @@ export class Watcher { .on("unlink", this.makeFileRemovedHandler()) .on("addDir", this.makeDirAddedHandler()) .on("unlinkDir", this.makeDirRemovedHandler()) + .on("ready", () => { + this.emit("ready") + }) + .on("all", (name, path, payload) => { + this.emit(name, path, payload) + this.log.silly(`FSWatcher event: ${name} ${path} ${JSON.stringify(payload)}`) + }) } this.processBuffer().catch((err: Error) => { diff --git a/garden-service/test/data/test-project-watch/.gardenignore b/garden-service/test/data/test-project-watch/.gardenignore index 418f6c60a9..ae84d0ee9b 100644 --- a/garden-service/test/data/test-project-watch/.gardenignore +++ b/garden-service/test/data/test-project-watch/.gardenignore @@ -1 +1 @@ -**/project-excluded.txt \ No newline at end of file +**/gardenignore-excluded.txt \ No newline at end of file diff --git a/garden-service/test/data/test-project-watch/garden.yml b/garden-service/test/data/test-project-watch/garden.yml index bf500f176a..56d715a634 100644 --- a/garden-service/test/data/test-project-watch/garden.yml +++ b/garden-service/test/data/test-project-watch/garden.yml @@ -1,5 +1,7 @@ kind: Project name: test-project-a +modules: + exclude: [module-a/project-excluded.txt] environments: - name: local providers: diff --git a/garden-service/test/data/test-project-watch/module-a/gardenignore-excluded.txt b/garden-service/test/data/test-project-watch/module-a/gardenignore-excluded.txt new file mode 100644 index 0000000000..fc21ffd88d --- /dev/null +++ b/garden-service/test/data/test-project-watch/module-a/gardenignore-excluded.txt @@ -0,0 +1 @@ +Nope! \ No newline at end of file diff --git a/garden-service/test/data/test-project-watch/module-a/new.txtfoo b/garden-service/test/data/test-project-watch/module-a/new.txtfoo new file mode 100644 index 0000000000..e69de29bb2 diff --git a/garden-service/test/data/test-projects/huge-project/.gitignore b/garden-service/test/data/test-projects/huge-project/.gitignore new file mode 100644 index 0000000000..c4bd802c01 --- /dev/null +++ b/garden-service/test/data/test-projects/huge-project/.gitignore @@ -0,0 +1 @@ +dir* \ No newline at end of file diff --git a/garden-service/test/data/test-projects/huge-project/README.md b/garden-service/test/data/test-projects/huge-project/README.md new file mode 100644 index 0000000000..a1f9b6fb0d --- /dev/null +++ b/garden-service/test/data/test-projects/huge-project/README.md @@ -0,0 +1,16 @@ +# Huge project tests + +Created for [PR #1320](https://github.com/garden-io/garden/pull/1320). + +This one's a bit hard to automate, but we can use to make sure Garden can handle massive amounts of files in repositories. + +The procedure to test was as follows: + +1) `cd` to this directory. +2) Run `node generate.js`. +3) Comment out the `modules.exclude` field in the `garden.yml`. +4) In `garden-service/src/watch.ts`, add `usePolling: true` to the options for the chokidar `watch()` function. +5) Run `garden build -w` and observe the process drain CPU and RAM until it crashes in about a minute. +6) Uncomment the `modules.exclude` field in the `garden.yml`. +7) Run `garden build -w` again, wait and observe a happy process for a while. +8) Run `rm -rf dir*` to clean up. diff --git a/garden-service/test/data/test-projects/huge-project/garden.yml b/garden-service/test/data/test-projects/huge-project/garden.yml new file mode 100644 index 0000000000..2a8bd67a22 --- /dev/null +++ b/garden-service/test/data/test-projects/huge-project/garden.yml @@ -0,0 +1,5 @@ +kind: Project +name: huge-project +dotIgnoreFiles: [.gardenignore] +modules: + exclude: [dir0/**/*, dir1/**/*, dir2/**/*, dir3/**/*, dir4/**/*, dir5/**/*, dir6/**/*, dir7/**/*, dir8/**/*] diff --git a/garden-service/test/data/test-projects/huge-project/generate.js b/garden-service/test/data/test-projects/huge-project/generate.js new file mode 100644 index 0000000000..6420d0e82a --- /dev/null +++ b/garden-service/test/data/test-projects/huge-project/generate.js @@ -0,0 +1,42 @@ +const { join } = require("path") +const { ensureDir, ensureFile } = require("fs-extra") + +let levels = 6 +let directoriesPerLevel = 6 +let filesPerLevel = 3 + +async function generateData(cwd, level) { + level++ + + let files = 0 + let directories = 0 + + for (let d = 0; d < directoriesPerLevel; d++) { + const dir = join(cwd, "dir" + d) + await ensureDir(dir) + directories++ + + for (let f = 0; f < filesPerLevel; f++) { + const file = join(dir, "file" + f) + await ensureFile(file) + files++ + } + + if (level < levels) { + const res = await generateData(dir, level) + files += res.files + directories += res.directories + } + } + + return { files, directories } +} + +generateData(process.cwd(), 0) + .then((res) => { + console.log(`Made ${res.files} files in ${res.directories} directories`) + }) + .catch(err => { + console.log(err) + process.exit(1) + }) diff --git a/garden-service/test/unit/src/watch.ts b/garden-service/test/unit/src/watch.ts index ce5b9d73f6..54b8c7a852 100644 --- a/garden-service/test/unit/src/watch.ts +++ b/garden-service/test/unit/src/watch.ts @@ -1,6 +1,6 @@ import { resolve, join } from "path" import { expect } from "chai" -import pEvent = require("p-event") +import pEvent from "p-event" import { TestGarden, @@ -12,12 +12,12 @@ import { makeExtProjectSourcesGarden, } from "../../helpers" import { CacheContext, pathToCacheContext } from "../../../src/cache" -import { createFile, remove, pathExists } from "fs-extra" +import { remove, pathExists, writeFile } from "fs-extra" import { getConfigFilePath } from "../../../src/util/fs" import { LinkModuleCommand } from "../../../src/commands/link/module" -import { cleanUpGlobalWatcher } from "../../../src/watch" import { LinkSourceCommand } from "../../../src/commands/link/source" import { sleep } from "../../../src/util/util" +import { Garden } from "../../../src/garden" function emitEvent(garden: TestGarden, name: string, payload: any) { garden["watcher"]["watcher"].emit(name, payload) @@ -43,25 +43,28 @@ describe("Watcher", () => { garden.events.clearLog() garden["watcher"]["addBuffer"] = {} garden["watcher"].start() + await waitUntilReady(garden) }) afterEach(async () => { - garden["watcher"].stop() + await garden["watcher"].stop() // Wait for processing to complete while (garden["watcher"].processing) { await sleep(100) } - - garden.events.clearLog() }) after(async () => { await garden.close() }) + async function waitUntilReady(_garden: Garden) { + return pEvent(_garden["watcher"], "ready", { timeout: 5000 }) + } + async function waitForEvent(name: string) { - return pEvent(garden.events, name, { timeout: 2000 }) + return pEvent(garden.events, name, { timeout: 5000 }) } it("should emit a moduleConfigChanged changed event when module config is changed", async () => { @@ -151,7 +154,7 @@ describe("Watcher", () => { it("if a file is added to a module", async () => { const path = resolve(modulePath, "new.txt") try { - await createFile(path) + await writeFile(path, "foo") expect(await waitForEvent("moduleSourcesChanged")).to.eql({ names: ["module-a"], pathsChanged: [path], @@ -174,6 +177,12 @@ describe("Watcher", () => { }) }) + it("should not emit moduleSourcesChanged if file is changed and matches the modules.exclude list", async () => { + const pathChanged = resolve(includeModulePath, "project-excluded.txt") + emitEvent(garden, "change", pathChanged) + expect(garden.events.eventLog).to.eql([]) + }) + it("should not emit moduleSourcesChanged if file is changed and doesn't match module's include list", async () => { const pathChanged = resolve(includeModulePath, "foo.txt") emitEvent(garden, "change", pathChanged) @@ -187,7 +196,7 @@ describe("Watcher", () => { }) it("should not emit moduleSourcesChanged if file is changed and it's in a gardenignore in the project", async () => { - const pathChanged = resolve(modulePath, "project-excluded.txt") + const pathChanged = resolve(modulePath, "gardenignore-excluded.txt") emitEvent(garden, "change", pathChanged) expect(garden.events.eventLog).to.eql([]) }) @@ -281,8 +290,6 @@ describe("Watcher", () => { const localModulePathB = join(localModuleSourceDir, "module-b") before(async () => { - // The watcher instance is global so we clean up the previous one before proceeding - cleanUpGlobalWatcher() garden = await makeExtModuleSourcesGarden() // Link some modules @@ -315,17 +322,10 @@ describe("Watcher", () => { // so the user will always have a new instance of Garden when they run their next command. garden = await makeExtModuleSourcesGarden() await garden.startWatcher(await garden.getConfigGraph()) - // This ensures that the watcher is properly initialised when we call `watcher.getWatched()` below - await sleep(100) - }) - - beforeEach(() => { - garden.events.clearLog() }) after(async () => { await resetLocalConfig(garden.gardenDirPath) - await garden.close() }) it("should watch all linked repositories", () => { @@ -353,8 +353,6 @@ describe("Watcher", () => { const localSourcePathB = join(localProjectSourceDir, "source-b") before(async () => { - // The watcher instance is global so we clean up the previous one before proceeding - cleanUpGlobalWatcher() garden = await makeExtProjectSourcesGarden() // Link some projects @@ -387,17 +385,10 @@ describe("Watcher", () => { // so the user will always have a new instance of Garden when they run their next command. garden = await makeExtProjectSourcesGarden() await garden.startWatcher(await garden.getConfigGraph()) - // This ensures that the watcher is properly initialised when we call `watcher.getWatched()` below - await sleep(100) - }) - - beforeEach(() => { - garden.events.clearLog() }) after(async () => { await resetLocalConfig(garden.gardenDirPath) - await garden.close() }) it("should watch all linked repositories", () => {