fix(watcher): use exclude options to optimize file watching (#1320)

* fix(watcher): use exclude options to optimize file watching

This basically relies on optimizations already implemented in chokidar,
the library we use for file watching. Hopefully this can solve the
excessive CPU/RAM usage some users have reported.

Closes #1269

docs/guides/configuration-files.md

@@ -353,6 +353,8 @@ Here we only scan the `modules` directory, but exclude the `modules/tmp` directo
 
 If you specify a list with `include`, only those patterns are included. If you then specify one or more `exclude` patterns, those are filtered out of the ones matched by `include`. If you _only_ specify `exclude`, those patterns will be filtered out of all paths in the project directory.
 
+The `modules.exclude` field is also used to limit the number of files and directories Garden watches for changes while running. Use that if you have a large number of files/directories in your project that you do not need to watch, or if you are seeing excessive CPU/RAM usage. The `modules.include` field has no effect on which paths Garden watches for changes.
+
 #### Module include/exclude
 
 By default, all files in the same directory as a module configuration file are included as source files for that module. Sometimes you need more granular control over the context, not least if you have multiple modules in the same directory.
@@ -375,6 +377,8 @@ Here we only include the `Dockerfile` and all the `.py` files under `my-sources/
 
 If you specify a list with `include`, only those files/patterns are included. If you then specify one or more `exclude` files or patterns, those are filtered out of the files matched by `include`. If you _only_ specify `exclude`, those patterns will be filtered out of all files in the module directory.
 
+Note that the module `include` and `exclude` fields have no effect on which paths Garden watches for changes. Use the project `modules.exclude` field (described above) for that purpose.
+
 #### .ignore files
 
 By default, Garden respects `.gitignore` and `.gardenignore` files and excludes any patterns matched in those files.

docs/reference/config.md

@@ -178,6 +178,10 @@ Note that you can also _exclude_ path using the `exclude` field or by placing `.
 source tree, which use the same format as `.gitignore` files. See the
 [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories) for details.
 
+Unlike the `exclude` field, the paths/globs specified here have _no effect_ on which files and directories
+Garden watches for changes. Use the `exclude` field to affect those, if you have large directories that
+should not be watched for changes.
+
 Also note that specifying an empty list here means _no paths_ should be included.
 
 | Type            | Required |
@@ -198,7 +202,26 @@ modules:
 [modules](#modules) > exclude
 
 Specify a list of POSIX-style paths or glob patterns that should be excluded when scanning for modules.
-Note that you can also explicitly _include_ files using the `include` field. If you also specify the `include` field, the paths/patterns specified here are filtered from the files matched by `include`. See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories) for details.
+
+The filters here also affect which files and directories are watched for changes. So if you have a large number
+of directories in your project that should not be watched, you should specify them here.
+
+For example, you might want to exclude large vendor directories in your project from being scanned and
+watched:
+
+```yaml
+modules:
+  exclude:
+    - node_modules/**/*
+    - vendor/**/*
+```
+
+Note that you can also explicitly _include_ files using the `include` field. If you also specify the
+`include` field, the paths/patterns specified here are filtered from the files matched by `include`.
+
+The `include` field does _not_ affect which files are watched.
+
+See the [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories) for details.
 
 | Type            | Required |
 | --------------- | -------- |
@@ -527,6 +550,10 @@ Note that you can also explicitly _include_ files using the `include` field. If
 `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the
 [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details.
 
+Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files
+and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have
+large directories that should not be watched for changes.
+
 | Type            | Required |
 | --------------- | -------- |
 | `array[string]` | No       |

docs/reference/module-types/container.md

@@ -103,6 +103,10 @@ Note that you can also explicitly _include_ files using the `include` field. If
 `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the
 [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details.
 
+Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files
+and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have
+large directories that should not be watched for changes.
+
 | Type            | Required |
 | --------------- | -------- |
 | `array[string]` | No       |

docs/reference/module-types/exec.md

@@ -106,6 +106,10 @@ Note that you can also explicitly _include_ files using the `include` field. If
 `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the
 [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details.
 
+Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files
+and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have
+large directories that should not be watched for changes.
+
 | Type            | Required |
 | --------------- | -------- |
 | `array[string]` | No       |

docs/reference/module-types/helm.md

@@ -98,6 +98,10 @@ Note that you can also explicitly _include_ files using the `include` field. If
 `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the
 [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details.
 
+Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files
+and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have
+large directories that should not be watched for changes.
+
 | Type            | Required |
 | --------------- | -------- |
 | `array[string]` | No       |

docs/reference/module-types/kubernetes.md

@@ -106,6 +106,10 @@ Note that you can also explicitly _include_ files using the `include` field. If
 `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the
 [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details.
 
+Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files
+and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have
+large directories that should not be watched for changes.
+
 | Type            | Required |
 | --------------- | -------- |
 | `array[string]` | No       |

docs/reference/module-types/maven-container.md

@@ -108,6 +108,10 @@ Note that you can also explicitly _include_ files using the `include` field. If
 `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the
 [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details.
 
+Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files
+and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have
+large directories that should not be watched for changes.
+
 | Type            | Required |
 | --------------- | -------- |
 | `array[string]` | No       |

docs/reference/module-types/openfaas.md

@@ -98,6 +98,10 @@ Note that you can also explicitly _include_ files using the `include` field. If
 `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the
 [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details.
 
+Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files
+and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have
+large directories that should not be watched for changes.
+
 | Type            | Required |
 | --------------- | -------- |
 | `array[string]` | No       |

docs/reference/module-types/terraform.md

@@ -112,6 +112,10 @@ Note that you can also explicitly _include_ files using the `include` field. If
 `include` field, the files/patterns specified here are filtered from the files matched by `include`. See the
 [Configuration Files guide](https://docs.garden.io/guides/configuration-files#including-excluding-files-and-directories)for details.
 
+Unlike the `modules.exclude` field in the project config, the filters here have _no effect_ on which files
+and directories are watched for changes. Use the project `modules.exclude` field to affect those, if you have
+large directories that should not be watched for changes.
+
 | Type            | Required |
 | --------------- | -------- |
 | `array[string]` | No       |

docs/troubleshooting.md

@@ -7,6 +7,12 @@ title: Troubleshooting
 
 _This section could (obviously) use more work. Contributions are most appreciated!_
 
+## I have a huge number of files in my repository and Garden is eating all my CPU/RAM
+
+This issue often comes up on Linux, and in other scenarios where the filesystem doesn't support event-based file watching.
+
+Thankfully, you can in most cases avoid this problem using the `modules.exclude` field in your project config, and/or the `exclude` field in your individual module configs. See the [Including/excluding files and directories](./using-garden/configuration-files#including-excluding-files-and-directories) section in our Configuration Files guide for details.
+
 ## I'm getting an "EPERM: operation not permitted, rename..." error on Windows
 
 This is a known issue with Windows and may affect many Node.js applications (and possibly others).

garden-service/package.json

@@ -36,7 +36,7 @@
     "certpem": "^1.1.3",
     "chalk": "^2.4.2",
     "child-process-promise": "^2.2.1",
-    "chokidar": "^3.0.2",
+    "chokidar": "^3.3.0",
     "ci-info": "^2.0.0",
     "circular-json": "^0.5.9",
     "cli-cursor": "^3.1.0",
@@ -218,7 +218,7 @@
     "e2e": "cd test/e2e && ../../bin/garden test",
     "e2e-project": "node build/test/e2e/e2e-project.js",
     "pkg": "./bin/build-pkg.sh",
-    "test": "node_modules/.bin/mocha",
+    "test": "node_modules/.bin/mocha --opts test/mocha.opts",
     "tsc": "tsc -p . --outDir build --declaration --declarationMap --sourceMap",
     "view-report": "open coverage/index.html",
     "watch": "npm run tsc -- -w"
@@ -230,4 +230,4 @@
     ]
   },
   "gitHead": "b0647221a4d2ff06952bae58000b104215aed922"
-}
+}

garden-service/src/config/module.ts

@@ -140,7 +140,12 @@ export const baseModuleSpecSchema = joi
 
         Note that you can also explicitly _include_ files using the \`include\` field. If you also specify the
         \`include\` field, the files/patterns specified here are filtered from the files matched by \`include\`. See the
-        [Configuration Files guide](${includeGuideLink})for details.`
+        [Configuration Files guide](${includeGuideLink})for details.
+
+        Unlike the \`modules.exclude\` field in the project config, the filters here have _no effect_ on which files
+        and directories are watched for changes. Use the project \`modules.exclude\` field to affect those, if you have
+        large directories that should not be watched for changes.
+        `
       )
       .example([["tmp/**/*", "*.log"], {}]),
     repositoryUrl: joiRepositoryUrl().description(

garden-service/src/config/project.ts

@@ -166,24 +166,47 @@ const projectModulesSchema = joi.object().keys({
     .array()
     .items(joi.string().posixPath({ subPathOnly: true }))
     .description(
-      dedent`Specify a list of POSIX-style paths or globs that should be scanned for Garden modules.
+      dedent`
+        Specify a list of POSIX-style paths or globs that should be scanned for Garden modules.
 
         Note that you can also _exclude_ path using the \`exclude\` field or by placing \`.gardenignore\` files in your
         source tree, which use the same format as \`.gitignore\` files. See the
         [Configuration Files guide](${includeGuideLink}) for details.
 
+        Unlike the \`exclude\` field, the paths/globs specified here have _no effect_ on which files and directories
+        Garden watches for changes. Use the \`exclude\` field to affect those, if you have large directories that
+        should not be watched for changes.
+
         Also note that specifying an empty list here means _no paths_ should be included.`
     )
     .example([["modules/**/*"], {}]),
   exclude: joi
     .array()
     .items(joi.string().posixPath({ subPathOnly: true }))
     .description(
-      deline`Specify a list of POSIX-style paths or glob patterns that should be excluded when scanning for modules.
+      dedent`
+        Specify a list of POSIX-style paths or glob patterns that should be excluded when scanning for modules.
+
+        The filters here also affect which files and directories are watched for changes. So if you have a large number
+        of directories in your project that should not be watched, you should specify them here.
+
+        For example, you might want to exclude large vendor directories in your project from being scanned and
+        watched:
+
+        \`\`\`yaml
+        modules:
+          exclude:
+            - node_modules/**/*
+            - vendor/**/*
+        \`\`\`
 
         Note that you can also explicitly _include_ files using the \`include\` field. If you also specify the
-        \`include\` field, the paths/patterns specified here are filtered from the files matched by \`include\`. See the
-        [Configuration Files guide](${includeGuideLink}) for details.`
+        \`include\` field, the paths/patterns specified here are filtered from the files matched by \`include\`.
+
+        The \`include\` field does _not_ affect which files are watched.
+
+        See the [Configuration Files guide](${includeGuideLink}) for details.
+      `
     )
     .example([["public/**/*", "tmp/**/*"], {}]),
 })

garden-service/src/garden.ts

@@ -256,7 +256,7 @@ export class Garden {
    * Clean up before shutting down.
    */
   async close() {
-    this.watcher && this.watcher.stop()
+    this.watcher && (await this.watcher.stop())
   }
 
   getPluginContext(provider: Provider) {