Expose details of module resolution

[ggoodman]

Now that incremental mode landed, I'm interested in implementing a sort of hybrid of nodemon and esbuild. I've hacked together an MVP that works very well for my use case.

However, it occurs to me that to generalize the solution, it would be important to be able to introspect which files are relevant to a given build. While we already have all the info about source files, two groups of files would be helpful to also see:

1. package.json files consulted in performing resolutions. Changes to these may invalidate parts of the build so an intelligent watcher would observe these for changes. Of course, a 100% bullet-proof solution would also need to observe certain directories where the absence of files with certain patterns like node_modules/<bare_module_target> or package.json might also invalidate resolutions that traversed up the filesystem.
2. tsconfig.json files consulted in performing resolutions and in configuring build settings. Similarly to the former, a watching process would want to watch the correct set of these to re-trigger incremental builds.

General solution idea

I've built a (much less ambitious or complete) bundler called Velcro that was designed to run as a service and to run in various js environments. Part of the design of Velcro involved defining a minimal set of operations that can define a module resolution interface across different conceptual stores (memory, fs, cdn like unpkg or even a collection of monaco-editor Models). Since reads to these could be potentially expensive, the idea was to think about caching and invalidation from the beginning.

To address this, the Velcro Resolver tracks a collection of path / access types that would invalidate the outcome of any of the cached operations. For example, if the resolver checks for the existence of file.ts before finding file.js via the default extensions setting, the creation of file.ts would invalidate that resolution.

Consumers of the Resolver, such as the Bundler receive the invalidation records for each operation and can remember these. When an event occurs that may invalidate something, all of the invalidations are pre-calculated. We just need to loop through what we have already tracked and delete records as well as tell the Resolver to delete its own: like this.

Anyway, all this to say that it would be interesting to think of how consumers of esbuild might be able to obtain some visibility into these resolution details so that they can be intelligent about when they trigger rebuilds. In general a naive watch pattern will work well enough but as soon as we get into weird directory structures and inter-linking tsconfig.json files, having this metadata would be almost essential.

[evanw]

This seems similar to these other comments of yours: #527 (comment) and #527 (comment).

I'm guessing the simplest way to solve for these and your other use cases would be to provide a way to hook into the file system abstraction itself. Luckily esbuild already has a file system abstraction internally, which is used for esbuild's default on-resolve behavior. That would come with a speed penalty of course, but you're already aware of that.

Hooking into the file system itself is a pretty advanced use case. I want to be thoughtful about how this is added or even if this should be added at all given that it's an advanced use case. For example, it's not clear to me if this use case is more of a "singleton" where it makes sense for there to just be one file system intercept or if there should be multiple ones for different purposes. Let's keep this issue open to track the possibility of replacing the file system.

For the record, the use cases you've brought up so far that are related to file system access are:

• Emulating a file system in the browser
• Learning about all of the files touched by esbuild for file watching

It's good that you're already aware of the subtlety that path resolution can be affected even if none of the input files from the previous build have changed. That case is easy to miss. There are other cases in addition to new node_modules directories popping up because of how complex path resolution is. For example, if your resolve extension order is .ts,.js and the previous build had a .js file, a newly-created .ts file could preempt the old .js file from the previous build. You can get even more complicated with arbitrary tsconfig.json path patterns such as "css!*.css" => "./styles/*.css" matching import paths with path components in the wildcard such as "css!../../dir/file.css". Cases like this are why esbuild's incremental build always re-runs all path resolution on every incremental build even if none of the source files have been changed.

[eulores]

I am also interested in emulating a file system in the browser. Right now, only natural files can be virtualized via the onLoad function defined in plugins.

Suggestion: Offer internal files, such as tsconfig.json and package.json to the onLoad function of installed plugins under an exclusive namespace (internal? esbuild?). If the plugin is interested in handling these, then it can return JSON data, and maybe signal a new dedicated content type for this. If not processed by the plugin, then the normal file resolution of esbuild takes care as before.

Files touched by esbuild should ideally be added to metafile, that way they can be indexed by file watchers.