[feat] expose esbuild as native NodeJS Node-API module to replace IPC with a zero overhead transfer for nodejs

[SalvatorePreviti]

On NodeJS, one option to improve the performance for transform operations or big bundles/source maps could be to not run esbuild as an external process but to load it as a DLL module with a thin C Node-API wrapper.

This would:

• reduce to zero the overhead added by the IPC communication (memory and time)
• reduce to zero or almost zero the cost of transferring memory between esbuild and js
• reduce to zero or almost zero the overhead of invoking js plugins
• allow to invoke the plugin also for synchronous build/transform
• allow to use the same esbuild instance between multiple worker threads
• allow the integration of esbuild with other languages without the IPC overhead

How could it be done:

1. expose esbuild as a static or dynamic library compiled with -buildmode=c-shared or -buildmode=c-archive, this would also potentially allow integration with C/C++/Rust/Python and other languages.

2. write a .node module in C/C++ that loads the static/shared library, starts a new thread for launching esbuild and executes the commands converting/adapting V8 data types to esbuild data types. Both sync and async methods. Change the makefile to compile both the executable and the .node module.

3. This .node native module could be published in a different set of packages per architecture - for example, esbuild-node-linux-64. Change install.js to understand if the running nodejs version is compatible.

4. modify the current nodejs loader library to use native .node module instead of the external process if available.

At the moment is just an idea but before trying to make this work, would be good to know if the esbuild team is interested in this. If this is something that the esbuild team would be interested in looking at for the future I would be more than happy to help/support as much as I can on this.

References:
https://nodejs.org/api/n-api.html
https://nodejs.org/api/addons.html
https://docs.google.com/document/d/1nr-TQHw_er6GOQRsF6T43GGhFDelrAP0NqSS_00RgZQ
https://medium.com/swlh/build-and-use-go-packages-as-c-libraries-889eb0c19838

[evanw]

This has been proposed and investigated before: #248. It's certainly possible. However, esbuild is primarily a bundler and the code transformation use case is secondary. I added support for code transformation because it was straightforward to add and useful for people but esbuild isn't designed to be optimal for that use case. I would have implemented esbuild differently if optimal language bindings was the primary goal; Go isn't a great choice for binding to other languages for a number of reasons.

My main hesitation with trying to change this is that it would introduce a lot more work for me. Right now esbuild supports 15 separate platforms with more on the way. This is currently trivial because Go has excellent cross-compilation support. I haven't had to set up individual CI instances for all of the various platforms because I basically haven't had any problems due to how Go's static linking works and the universality of stdin/stdout IPC. The main cross-platform problems I've had are with Windows paths, which is an issue regardless.

Doing what you are suggesting here would be a big step up in complexity for releases and I suspect would require more complexity on the CI side as well. It would also have to be repeated for each of the 15 platforms that esbuild supports. And it would have to be maintained which would mean ongoing work as platforms inevitably encounter build issues. Ideally it would be possible to build everything locally instead of relying on a fleet of CI machines to do the builds (right now I can build esbuild on all platforms in a few seconds, so releases are trivial). But getting cross-compilation to run smoothly can also be a lot of work, especially since C/C++ build tools are obscenely huge and take hours to build for each platform. I've heard that the Zig toolchain is good at cross-compilation so maybe that would help. That would have to be investigated.

TL;DR: I'm not keen on doing this since it seems like a lot of work and it's not even the main point of esbuild anyway. I mainly designed esbuild to set the bar for how long a I think creating a production bundle should take. I hope esbuild's speed inspires others to create code transformation libraries (probably not written in Go) that allow for high-performance language bindings, but at the moment I'm not interested in maintaining something like that myself. People should still feel free to experiment with getting this to work with esbuild if they'd like to but I'm not going to give any guarantees on accepting such patches.