Incremental builds using .rebuild. Doesn't ignore unchanged files?

[dbousamra]

I am currently building out my dev environment using esbuild. My build currently takes 6s. 5s of that are in my Less plugin.

Even when I change just the Typescript React code, it still appears to build the Less files.

Is this expected behaviour? Is there a way to tell esbuild to only build what has changed?

Here is my less plugin https://github.com/Piccolo-Health/esbuild-plugin-less

[nettybun]

If less files take a very long time to build would it work for you to have a separate watcher/builder for *.less files that outputs to a directory, and then have your esbuild plugin simply redirect *.less imports to their respective css file?

That way esbuild doesn't even track or build less files, so it's safe and fast to edit TS React code

[evanw]

Plugins cannot be cached by esbuild since it has no insight into what the plugin does. It could be returning `/* ${new Date} */` for all esbuild knows. More realistically, it could be accessing arbitrary files on the file system, in which case it would be incorrect to cache the result based on just that one file given to the plugin because then the result could be stale if one of the other files accessed by the plugin was changed (e.g. if the plugin processes @import directives itself).

You will have to implement caching for your plugin yourself, inside your plugin. I don't know about the details of your plugin but if the output only depends on that one input file, you can use an in-memory map to cache the transform. Essentially you just memoize the transform function. This will work with incremental builds because your plugin's memory stays around in between builds.

[dbousamra]

If less files take a very long time to build would it work for you to have a separate watcher/builder for *.less files that outputs to a directory, and then have your esbuild plugin simply redirect *.less imports to their respective css file?

That way esbuild doesn't even track or build less files, so it's safe and fast to edit TS React code

Pragmatically, that's a fine solution to get my project working with esbuild.

Plugins cannot be cached by esbuild since it has no insight into what the plugin does. It could be returning /* ${new Date} */ for all esbuild knows. More realistically, it could be accessing arbitrary files on the file system, in which case it would be incorrect to cache the result based on just that one file given to the plugin because then the result could be stale if one of the other files accessed by the plugin was changed (e.g. if the plugin processes @import directives itself).

Thanks, this is a good explanation, and makes sense to me.

You will have to implement caching for your plugin yourself, inside your plugin. I don't know about the details of your plugin but if the output only depends on that one input file, you can use an in-memory map to cache the transform. Essentially you just memoize the transform function. This will work with incremental builds because your plugin's memory stays around in between builds.

Okay, I will look at implementing some sort of caching mechanism. Is the architectural idea of plugins caching, something you see as a good thing for esbuild long term, and how does it fit with your end goals? As an aside, I am loving the experience with esbuild so far. It's been really enjoyable to use.

[evanw]

Is the architectural idea of plugins caching, something you see as a good thing for esbuild long term, and how does it fit with your end goals?

Caching is great! Ideally people writing plugins for esbuild would keep performance in mind and caching is part of that.

You may be wondering why esbuild doesn't have a built-in caching mechanism. While memoizing a pure function is pretty simple, cache invalidation in other scenarios can get very complicated and custom, so it would be error-prone for esbuild to try to infer caching rules itself. There are some examples of cache invalidation complexities in #555 (comment).

The only sort-of-automatic way of doing correct cache invalidation that I know of is intercepting syscalls for the current process and all child processes to look for file system accesses. But even then that would fail to capture other forms of non-determinism such as dates, random numbers, or network requests. Doing these intercepts may also not be portable to all platforms that esbuild supports (I don't know because I haven't tried). It's also likely incompatible with esbuild's current API, which is run from within another process as a library.

Basically the plugin author knows more than esbuild about how to do cache invalidation for their plugin.

[dbousamra]

Alright, great.

You may be wondering why esbuild doesn't have a built-in caching mechanism. While memoizing a pure function is pretty simple, cache invalidation in other scenarios can get very complicated and custom, so it would be error-prone for esbuild to try to infer caching rules itself. There are some examples of cache invalidation complexities in #555 (comment).

Absolutely. I think it makes sense for esbuild to not worry about this. Let the community and plugin authors sort performance out. It gives people room for optimization and keeps esbuild core slimmer.

[evanw]

Closing this since it sounds like the question was answered.