Support the esbuild plug-in system？ #111
Comments
|
Not right now. I may figure out an extensibility story within esbuild after the project matures, but I may also keep esbuild as a relatively lean bundler with only a certain set of built-in features depending on how the project evolves. The use case of using esbuild as a library was one I hadn't originally considered. It's interesting to see it start to take off and I want to see where it goes. It could be that most users end up using other bundlers and esbuild is just an implementation detail that brings better performance to those bundlers. I'm still thinking about how I might add extensibility to esbuild in the back of my mind. Obviously it's made more complicated by the fact that it's written in Go. It would be possible to "shell out" to other processes to delegate the task of transforming input files, but that would almost surely be a huge slowdown because JavaScript process startup overhead costs are really high. One idea I've been thinking about is to have esbuild start up a set number of JavaScript processes (possibly just one) and then stream commands to it over stdin/stdout. That could potentially amortize some of the JavaScript startup cost (loading and JITing packages from disk). It would be more of a pain to debug and might still be surprisingly slow due to all of the serialization overhead and the single-threaded nature of the JavaScript event loop. Another idea is to turn the esbuild repository into a Go-based "build your own bundler" kit. Then you could write plugins in Go to keep your extensions high-performance. The drawback is that you'd have to build your own bundler, but luckily Go compiles quickly and makes cross-platform builds trivial. That would likely require me to freeze a lot of the Go APIs that are now internal-only, which would prevent me from making major improvements. So this definitely isn't going to happen in the short term since esbuild is still early and under heavy development. |
|
Finding this interesting! TL;DR: The subprocess approach sounds like a solid idea. Some thoughts in regards to the conversation above
This is a really interesting approach I think, although not the most portable (i.e. running systems like iOS that do not support subprocess creation.) I'd think about the quality-performance problem here in the same way as with Figma plugins:
In my experience what makes nodejs programs slow is I/O rather than CPU. For example, loading gatsby.js causes an incredible amount of files and directories to be read etc. TypeScript is an example of a good player making the best they can, avoiding runtime imports, but starting tsc is still painfully slow (thus their daemon/server model, which is a solution just like the "subprocess" idea you have!)
Perhaps a nice option for people who are comfortable with Go. Might "pair well" with a subprocess approach for "putting lego blocks together" vs "build your own lego blocks".
Some thoughts on Go pluginsI've worked with plugins in go in the past and it's a little bit complicated (for good reasons.) Since Go is statically compiled and doesn't have a fully dynamic runtime like for example JavaScript it is a little tricky to load code at runtime and really hard to unload code (replace/update.) Some things to keep in mind:
Some example code from GHP: Loading plugins: Add structure to plugins to allow unloading them (without actually unloading their code.) A plugin: (called "servlet" in this project) |
|
What about something like this https://github.com/hashicorp/go-plugin. Use RPC based plugin system so anyone could write plugin in any language. |
|
@rsms thanks for writing up your thoughts. It was very interesting to read through them, and helpful to learn from your experience. I haven't seen the
Yes, I was thinking of something extremely minimal. Of course that would come at the cost of performance, which isn't great. I'm not sure if there's a great solution to this.
I think something like this is promising. This is basically how esbuild's current API works, except over stdin/stdout. The advantage of this over shelling out is that it lets you amortize the startup overhead of node by keeping it running during the build. You could imagine a more complex API where esbuild has hooks for various points and could call out to node and block that goroutine on the reply. That would let you, say, run the CoffeeScript compiler before esbuild processes the source code of a file. I plan to explore this direction once esbuild is more feature-complete. My research direction is going to be "assuming you have to use a JavaScript plugin, how fast can you make it". If we can figure that out then that's probably the most helpful form of API for the web development community. |
|
I would like to vote for this feature. Currently, I'm migrating some small project from Thank you |
|
If you do implement a plugin system, please consider making it Go-based (or better yet, cross-language as per @zmitry's suggestion). I think there is a very big opportunity for non-JS-based tooling to transpile JS. As you state in your readme:
|
|
I guess for first iteration it would be nice to have just golang api for plugins. I guess if you can run some arbitrary code in golang you could spawn sub process in any language. So even simple golang api would be enough. |
|
I have an update! While the final plugin API might need a few rewrites to work out the kinks, I think I have an initial approach that I feel pretty good about. It reuses the existing stdio IPC channel that the JavaScript API is already using and extends it to work with plugins. Everything is currently on an unstable branch called It's still very much a work in progress but I already have loader plugins working. Here's what a loader plugin currently looks like: let esbuild = require('esbuild')
let YAML = require('js-yaml')
let util = require('util')
let fs = require('fs')
esbuild.build({
entryPoints: ['example.js'],
bundle: true,
outfile: 'out.js',
plugins: [
plugin => {
plugin.setName('yaml-loader')
plugin.addLoader({ filter: /\.ya?ml$/ }, async (args) => {
let source = await util.promisify(fs.readFile)(args.path, 'utf8')
try {
let contents = JSON.stringify(YAML.safeLoad(source), null, 2)
return { contents, loader: 'json' }
} catch (e) {
return {
errors: [{
text: (e && e.reason) || (e && e.message) || e,
location: e.mark && {
line: e.mark.line,
column: e.mark.column,
lineText: source.split(/\r\n|\r|\n/g)[e.mark.line],
},
}],
}
}
})
},
],
}).catch(() => process.exit(1))Any errors during loading are integrated into the existing log system so they look native. There is a corresponding Go API that looks very similar. In fact the JavaScript plugin API is implemented on top of the Go plugin API. The API consists of function calls with option objects for both arguments and return values so it should hopefully be easy to extend in a backwards-compatible way. Loader plugins are given a module path and must come up with the contents for that module. I'm going to work on resolver plugins next which determine how an import path maps to a module path. Resolver plugins and loader plugins go closely together and many plugins are probably going to need both a resolver and a loader. The resolver runs for every import in every file while the loader only runs the first time a given resolved path is encountered. Something that may be different with this plugin API compared to other bundlers is that every operation has a filter regular expression. Calling out to a JavaScript plugin from Go has overhead and the filter lets you write a faster plugin by avoiding unnecessary plugin calls if it can be determined using the regular expression in Go that the JavaScript plugin isn't needed. I haven't done any performance testing yet so I'm not sure how much slower this is, but it seemed like a good idea to start things off that way. One weird thing about writing plugins is dealing with two forms of paths: file system paths and "virtual paths" to automatically-generated code. I struggled with the design of this for a while. One approach is to just use absolute paths for everything and make up non-existent directories to put virtual modules in. That leads to concise code but seems error-prone. Another approach I considered was to make every path into a tuple of a string and a type. That's how paths are represented internally but seemed too heavy for writing short plugins. I'm currently strongly considering marking virtual paths with a single null byte at the front like Rollup convention. Null bytes make the path invalid and the code for manipulating them is more concise than tuple objects. I thought it'd be a good idea to post an update now even though it's not quite ready to try out, since getting loaders working seemed like an important milestone. |
|
After more thought, I'm no longer thinking of taking the approach Rollup does with virtual paths using a null byte prefix. Instead I'm going back to the "paths are a tuple" model described above. In the current form, each path has an optional Also, I just got resolver plugins working! This lets you intercept certain paths and prevent the default resolver from running. Here's an example of a plugin that uses this to load URL imports from the network: // import value from 'https://www.google.com'
let https = require('https')
let http = require('http')
let httpLoader = plugin => {
plugin.setName('http')
plugin.addResolver({ filter: /^https?:\/\// }, args => {
return { path: args.path, namespace: 'http' }
})
plugin.addLoader({ filter: /^https?:\/\//, namespace: 'http' }, async (args) => {
let contents = await new Promise((resolve, reject) => {
let lib = args.path.startsWith('https') ? https : http
lib.get(args.path, res => {
let chunks = []
res.on('data', chunk => chunks.push(chunk))
res.on('end', () => resolve(Buffer.concat(chunks)))
}).on('error', reject)
})
return { contents, loader: 'text' }
})
}The resolver moves the paths to the Plugins can generate arbitrarily many virtual modules by importing new paths and then intercepting them. Here's a plugin I made to test this feature that implements the Fibonacci sequence using modules: // import value from 'fib(10)'
let fibonacciLoader = plugin => {
plugin.setName('fibonacci')
plugin.addResolver({ filter: /^fib\((\d+)\)/ }, args => {
return { path: args.path, namespace: 'fibonacci' }
})
plugin.addLoader({ filter: /^fib\((\d+)\)/, namespace: 'fibonacci' }, args => {
let match = /^fib\((\d+)\)/.exec(args.path), n = +match[1]
let contents = n < 2 ? `export default ${n}` : `
import n1 from 'fib(${n - 1}) ${args.path}'
import n2 from 'fib(${n - 2}) ${args.path}'
export default n1 + n2`
return { contents }
})
}Importing from the path |
|
The examples are impressive, and the I'm wondering if you could give a little more context regarding usage? From trying to sort through the |
|
Yes, good point. The plugin API is intended to be used with the esbuild API. People have already been creating simple JavaScript "build script" files that just call the esbuild API and exit. This is a more convenient way of specifying a lot of arguments to esbuild than a long command line in a package.json script. From there you can use plugins by just passing an additional const { build } = require('esbuild')
let envPlugin = plugin => {
plugin.setName('env-plugin')
plugin.addResolver({ filter: /^env$/ }, args => {
return { path: 'env', namespace: 'env-plugin' }
})
plugin.addLoader({ filter: /^env$/, namespace: 'env-plugin' }, args => {
return { contents: JSON.stringify(process.env), loader: 'json' }
})
}
build({
entryPoints: ['entry.js'],
bundle: true,
outfile: 'out.js',
plugins: [
envPlugin,
],
}).catch(() => process.exit(1))In reality I assume most of these plugins will be in third-party packages maintained by the community, so you would likely import the plugin using The Go API is extremely similar. Here's the same example using the Go API instead: package main
import (
"encoding/json"
"io/ioutil"
"log"
"os"
"path/filepath"
"strings"
"github.com/evanw/esbuild/pkg/api"
)
func main() {
result := api.Build(api.BuildOptions{
EntryPoints: []string{"entry.js"},
Bundle: true,
Write: true,
LogLevel: api.LogLevelInfo,
Outfile: "out.js",
Plugins: []func(api.Plugin){
func(plugin api.Plugin) {
plugin.SetName("env-plugin")
plugin.AddResolver(api.ResolverOptions{Filter: "^env$"},
func(args api.ResolverArgs) (api.ResolverResult, error) {
return api.ResolverResult{Path: "env", Namespace: "env-plugin"}, nil
})
plugin.AddLoader(api.LoaderOptions{Filter: "^env$", Namespace: "env-plugin"},
func(args api.LoaderArgs) (api.LoaderResult, error) {
mapping := make(map[string]string)
for _, item := range os.Environ() {
if equals := strings.IndexByte(item, '='); equals != -1 {
mapping[item[:equals]] = item[equals+1:]
}
}
bytes, _ := json.Marshal(mappings)
contents := string(bytes)
return api.LoaderResult{Contents: &contents, Loader: api.LoaderJSON}, nil
})
},
},
})
if len(result.Errors) > 0 {
os.Exit(1)
}
}Plugins aren't designed to be used on the command line. This is the first case of the full API not being available from the command line, but given that plugins are language-specific I think it makes sense to require you to use the language-specific esbuild API to use plugins. |
|
It would be useful to have a transform hook where you gain access to the generated AST, so that you can do fast file transformations in Go. |
|
Excellent examples, thanks!
Okay got it, yeah this seems wise. |
I totally understand why this would be useful, but I don't want to expose the AST in its current form. It's designed for speed, not ease of use, and there are lots of subtle invariants that need to be upheld (e.g. scope tree, symbol use counts, cross-part dependency tracking, import and export maps, ES6 import/export syntax flags, ordering of lowering and mangling operations, etc.). Exposing this internal AST to plugins would be a good way to destabilize esbuild and cause silent and hard-to-debug correctness issues with the generated code. I'm also trying to keep the quality of esbuild high, both in terms of the user experience and the developer experience. I don't want to expose the internal AST too early and then be stuck with that interface, since I don't think it's the right interface. Figuring out a good interface for the AST that is easy to use, doesn't slow things down too much, and hard to cause code generation bugs with would be a good project to explore. But this is a big undertaking and I don't think now is the right part in the timeline of this project to do this. It also makes a lot of other upcoming features harder (e.g. code splitting, other file types such as HTML and CSS) because it freezes the AST interface when it might need to change. For now, it's best to either serialize the AST to a string before passing it to esbuild or use other tools if you need to do AST manipulation. |
|
CSS extraction was actually pretty simple in Go: package main
import (
"bytes"
"io"
"os"
"github.com/evanw/esbuild/pkg/api"
)
var cssExport = "export default {};\n"
// CSSExtractor will accumulate CSS into a buffer.
type CSSExtractor struct {
bytes.Buffer
}
// Plugin can be used in api.BuildOptions.
func (ex *CSSExtractor) Plugin(plugin api.Plugin) {
plugin.SetName("css-extractor")
plugin.AddLoader(
api.LoaderOptions{Filter: `\.css$`},
func(args api.LoaderArgs) (res api.LoaderResult, err error) {
f, err := os.Open(args.Path)
if err != nil {
return res, err
}
defer f.Close()
if _, err := io.Copy(ex, f); err != nil {
return res, err
}
// CSS is an empty export.
res.Loader = api.LoaderJS
res.Contents = &cssExport
return res, nil
},
)
}This works for me, since I just want to write my CSS to a file. |
|
Is my understanding correct that this will then require a “wrapper” around esbuild in either go or node (or another language that implements the protocol node is using), and plugins will have to be written it that language? I.e. you wont be able to run esbuild --plugin download --plugin somethingelse, and have them be written in whatever? In other words more than starting an ecosystem of plugins for esbuild, likely a wrapper tool will emerge in both languages and plugin ecosysytems for the wrappers? |
|
I'm expecting all serious usage of esbuild to use the API anyway because specifying a long list of options on the command line isn't very maintainable (e.g. don't get nice diffs or git blame). You can easily do this without a separate "wrapper" package just by calling esbuild's JavaScript API from a file with a few lines of code: const { build } = require('esbuild')
build({
entryPoints: ['./src/main.ts'],
outfile: './dist/main.js',
minify: true,
bundle: true,
}).catch(() => process.exit(1))From that point, adding plugins is just adding another property to the build call. I'm sure some people will create fancy wrappers but a wrapper isn't necessary to use plugins. I'm also expecting that the large majority of esbuild plugins will be JavaScript plugins. Virtually all of the plugins in the current bundler community are written in JavaScript and people likely won't rewrite them when porting them to esbuild. So my design for plugins is primarily oriented around JavaScript and its ecosystem, not around Go. In that world most people wouldn't need a wrapper. As far as non-JavaScript languages, that stuff can get extremely custom and I think exposing a general API like the current Go API is better than trying to guess up front what people would want in a native language wrapper and hard-coding that into esbuild. You should be able to use the Go API to do whatever custom native language bindings you want (local sockets, child processes, RPC with a server, etc.) without any performance overhead over what esbuild would have done itself anyway. |
|
hello 👋🏼 just wanna add our use case here for the AST plugin API. I maintain react-intl/formatjs and we do have babel plugin & TS AST transformer to precompile ICU messages since runtime parsing is pretty costly. Therefore we'd love to support esbuild as well. |
|
Would it be possible to add (async) hooks for build start and build end? I'm thinking similar to rollups buildStart and buildEnd hooks. Right now we have |
|
I can see needing async I have already been planning to add callbacks for |
|
That sounds great. For my current use case I was planning on making the inputOptions be derived from a file on disk, and so it would be nice if it was possible to return Also I’m wondering at what point you imagine |
That is incorrect. The
I plan for it to get the final |
That sounds good. Is the plan to have it so that the build result can be mutated via this hook? Also I am wondering if these two use cases I have would be able to be tackled via one of these new build hooks or what your thoughts are.
|
|
@evanw any reason why the onResolve of a plugin executed sequentially for every import from one file? I'm writing my own bundler on top of esbuild and I need to resolve every single import via a plugin. When I tried to bundle |
Is this your final decision? I'm wanting to switch to Vite, but as mentioned above, FormatJS requires an AST. 😞 |
The vast majority of source files only have a few imports, so this has never come up. Having over 5,000 imports in one file is an extreme edge case. Parallelizing things can have overhead so you have to be careful when adding more parallelism. I'm willing to parallelize this if it's an improvement for this edge case if and only if it doesn't regress performance for common scenarios at all.
Yes. You are still welcome to transform the JavaScript code with other tools before or after esbuild runs of course. |
|
|
|
When not bundling, plugin system is not invoked at all. In my use-case I want to modify some import specifiers from our internal convention to something that works in node runtime. A simple plugin does the job but I noticed that the plugin's |
|
I see the comment above about no AST, but is there any possibility for a plugin hook that allows a custom JS transformations to be applied? It doesn't need to be handed the AST. Just the final JS after all the transformations that esbuild has applied. The hook is given that JS (and sourcemap) and whatever JS it returns (possibly with an updated sourcemap) is included in the bundle (or passed onto the next plugin that also adds a transformation). This way if the output of a plugin is JavaScript code and indicates the JS loader then the transformation can be applied even if the original source was not a JS file (for example a Vue component). My need here is driven from the trying to add code coverage instrumentation via Istanbul. I can currently use the esbuild-babel plugin in combination with the babel-plugin-istanbul plugin. But since esbuild-babel hooks into onLoad it only see the JS on disk not any JS generated from another plugin (like the esbuild-vue plugin). If such a hook existed to add a custom transformation then esbuild-babel could be modified to operate on this new hook instead of onLoad and therefore could apply to all JS generated by another plugin and not just JS on disk. For more background (as well as my hack workaround) see marvinhagemeister/karma-esbuild#33 (comment) |
|
#647 proposes onTransform hooks that runs after onLoad but before any transformation by esbuild itself |
|
@evanw, Is there a way for a JS plugin to further transform the code after it has been transformed by esbuild itself, but before the file is finally written to disk? I need to make some changes to the code after it has been converted by esbuild from TypeScript to JS. Thank you! |
|
@ZimNovich you could set the (and then you'll have to write them to disk yourself) |
|
@ZimNovich Check out the Transform API. Something along these lines will do the trick: const plugin = {
setup(build) {
build.onLoad({ filter: /\.svelte$/ }, async (args) => {
const source = await fs.promises.readFile(args.path, 'utf8');
const { code, map } = this.build.esbuild.transformSync(source, {
loader: "ts",
});
const contents = code; // manipulated code
return { contents }
})
}
} |
|
Jumping in late to the party here. If not, will it be possible to gain access to the build graph? If so I could use the webpack runtime as a template string and reconnect the graph references to esbuild (bit of a hack) |
|
I'm closing this issue as esbuild has had a stable plugin API for quite a while now, and I'm not currently planning on making any big/fundamental changes to the plugin API. Ongoing plugin API improvements are tracked by other issues. |
|
@evanw https://esbuild.github.io/faq/#upcoming-roadmap
Thank YouThanks for the awesome lib, I now use it in a few of my repos. 🚀 |
and others
esbuildis small but complete in every detail, do you have any plans to support the plugin-in system to extend the web development workflows?The text was updated successfully, but these errors were encountered: