File watching optimization

[matuszeman]

Garden seems to watch all files under project folder (ref: garden.ts#L287) which are then filtered out depending on e.g. module settings (ref: watch.ts#L172). This causes an issue with larger projects and the process runs out of memory after a while.

Could we discuss best way how this could be optimized please?

[matuszeman]

Here are some options which came to my mind:

• Add new configuration options watchPaths: string[] on project/module level.
• Use existing project module include/exclude and module include/exclude configuration.

[edvald]

Thanks a lot for filing the issue @matuszeman. I've done a little bit of exploration on the matter.

We can technically introduce an optimization when modules.include is specified in the project config, and includes only specific directories (i.e. some/directory/**/*). But unfortunately any more open-ended pattern in that list (e.g. *.js or anything else without a specific directory path) would require us to look at the whole project and post-filter, like we currently do. So users would need to be quite specific for us to be able to avoid watching the whole project root, and I'm not sure that's reasonable in terms of user experience.

If you specify exclude filters, there's still really no way for us to avoid watching the whole project root, because the list of directories to watch can always change, so we couldn't be aware of new directories if they're not already under the watched directories when we start the Garden process.

So I think your first suggested option is the only viable one to solve your issue (unless I'm missing a better idea, which I won't rule out :P). By default I resist adding config options, but I don't really see a way around it here, since the include/exclude filters can't sufficiently inform us what to watch except in very specific cases.

@matuszeman would you like to have a go at implementing a watchPaths option, or would you like to wait until we find time to add it? We'd love to help you contribute if you're up for it :)

[matuszeman]

@edvald I played with it yesterday a bit, it's not tested at all, I don't even know if this approach helps the problem. I'd need to run it on the project I had a issue with before. Have a look.

I wanted to stay consistent with other include / exclude configuration options garden already has and I think it makes sense to leverage chodikar ignored option for exclude here.