esbuild-wasm transform API

[baryla]

I've checked out the demo of esbuild-wasm that you have provided (try.html) and noticed this it covers the build scenario. Using esbuild-wasm, is it possible to use the transform API without the FS stubbing etc which build requires? It may be a little tricky with as we currently have to start the esbuild service.

I have a use case where I want to transform a single file at a time. It will have imports but it won't actually import any file to the bundle. I'll be handling that myself outside of esbuild.

Thanks for the this amazing tool which is moving the web forward!

[kzc]

Both the CLI and transform API work with esbuild-wasm. Just replace require('esbuild') with require('esbuild-wasm') in the transform example.

[baryla]

Thanks for the message @kzc. I am aware that both builds (esbuild and esbuild-wasm) have the same underlying API's exported.

Maybe I should've made my question more clear. The example that I mentioned in the first message uses the WASM version in a browser environment. To get access to the transform API, we need to start a service which uses child_process which is of course not supported in the browser (maybe we could stub it with WebWorkers instead).

I was wondering if it's even possible to use the transform API in browser, similar to how the example uses build. If it is - I would be grateful for some pointers about how to get that to work and what needs stubbing. It could be a stupid question as I'm not super familiar with WebAsssembly.

[kzc]

It looks like lib/main.js could be ported to a browser target with a little effort. I think the necessary FS stubbing for the go runtime would only affect the worker thread that esbuild-wasm would be running on.

[evanw]

The reason why the browser API isn't documented is that it's not intended for general use. It's just something I use during development to try out esbuild interactively to search for bugs.

It's currently a giant hack. Not only is the file system emulation thing unnecessary and awkward, but the current browser version also recreates the entire WebAssembly module every time you build. As you can imagine this makes it extremely slow.

This code was written before the more efficient service-oriented API was added. I plan to replace it with the service-oriented API before documenting it as ready for use.

@baryla Can you say more about what you're trying to use esbuild in the browser for?

[baryla]

That's fair @evanw. Thanks for the response.

Background
I'm currently working on a rewrite of Packager - a tool that aims to greatly simplify building online JS playgrounds. Packager is essentially an API which uses Web Workers and transpilers (such as sucrase) under the hood to provide a JS bundle for any type of project (Vue, React etc). All of this happens in the browser.

The current implementation uses sucrase for JS and TS transpiling which is pretty good but it lacks some of the functionality and speed that esbuild brings to the table. The thing that I find super exciting in esbuild (among other things of course) is the CommonJS <> ESM handling. Packager has automatic NPM resolution and unfortunately, most of the packages are still CommonJS so the fact that esbuild takes care of this is absolutely amazing - this is something that other tools don't provide.

We are currently seeing a push for online playgrounds (Codesandbox, Codespaces etc.) from the JS community so I think Packager would help in this movement. Current version of Packager has an average build time of ~50ms in a TypeScript project. With esbuild - I think we could go quicker and reduce this to <25ms to make the bundling truly instant.

The way I'd love to use esbuild
As I briefly mentioned - Packager uses Web Workers to offload the heavy transpilation/bundling process to the worker thread. Most of the tools that Packager uses, have very basic transform APIs that achieve exactly what I need eg: {tool}.transform(source, options). These tools are probably slightly less intensive than esbuild so I totally understand that starting the whole transform process per file would be crazy and the idea of a "running" service is important to avoid just that.

I think many tools, including Packager would benefit from being able to use esbuild in a Web Worker like this:

importScripts("HOSTED_ESBUILD");

const service = esbuild.startService();

self.addEventListener('message', async ({ data: { file } }) => {
  const options = {};
  const transpiled = await service.transform(file, options);

  self.postMessage(transpiled);
});

In the above example - esbuild isn't aware of a file system or anything other than "transpile this piece of code for me and don't worry about anything else". Packager handles everything else like imports, caching etc.

I hope this makes sense :)

[evanw]

Yes that makes sense. Thanks for the information.

I plan to port the service-oriented API to the browser at some point, which should address what you need.

[baryla]

That's amazing news! Looking forward to it @evanw 🚀

[evanw]

As of version 0.5.0, the esbuild-wasm package now has a transform API that works in the browser. This is documented here. Please let me know what you think about the API.

It would also be interesting to hear how performance compares to what you have been using previously. I haven't done any performance measurements of the browser API myself yet.

[baryla]

Amazing! I'm super excited about this! 🎉

I'll take a proper look at the API today and I'll try to integrate it into Packager and test it over the weekend and let you know how it performs, my thoughts etc.

Thanks for your work.

[baryla]

I just ran a quick demo against Packager and unfortunately, I'm running into some issues.

Here's the code that I used for the demo:

self.exports = {};
self.esbuildService = {};

const loadEsbuildService = async () => {
  if (!self.esbuildService) {
    self.importScripts(
      "https://cdn.jsdelivr.net/npm/esbuild-wasm/lib/browser.js"
    );

    self.esbuildService = await startService({
      wasmURL: "https://unpkg.com/browse/esbuild-wasm/esbuild.wasm",
    });
  }
}

self.addEventListener('message', async ({ data: { file } }) => {
  await loadEsbuildService();

  const transpiledCode = await self.esbuildService.transform(file.code, {
    target: "es2015",
    loader: "ts",
  });

  self.postMessage({
    code: transpiledCode
  });
});

The above code is a snippet of esbuild inside a WebWorker.

Here's a few points:

• Importing esbuild-wasm browser API from a CDN fails without self.exports = {}; or window.exports = {} because exports is not defined. Maybe a UMD build would be better to cover this case if we're not using require?
• I would recommend adding the API's under a global like esbuild instead of adding them directly to the target so we can access the API's like this: esbuild.startService. This way, we won't have any possible name clashes with top level properties.

And lastly - the above code produces an error message of:

CompileError: WebAssembly.instantiate(): expected magic word 00 61 73 6d, found 3c 21 44 4f @+0

I'm unfamiliar with WebAssembly to figure out what is causing this error, unfortunately.

[evanw]

Importing esbuild-wasm browser API from a CDN fails without self.exports = {}; or window.exports = {} because exports is not defined. Maybe a UMD build would be better to cover this case if we're not using require?

The file is in CommonJS format and is intended to be bundled with a bundler. Are you not using a bundler because the file is too big? I can cut the size down to 13kb by minifying it (5kb gzipped).

I'm unfamiliar with WebAssembly to figure out what is causing this error, unfortunately.

This is saying that the first four bytes are 3c 21 44 4f which is <!DO. This means you got the URL wrong and the browser is serving you HTML instead of WebAssembly. If you load https://unpkg.com/browse/esbuild-wasm/esbuild.wasm you get a HTML page that links to the correct URL: https://unpkg.com/[email protected]/esbuild.wasm.

Independent of the changes described above, I think this code should work (assuming importScripts can load scripts synchronously):

self.exports = {};

const loadEsbuildService = async () => {
  if (!self.esbuildService) {
    self.importScripts(
      "https://cdn.jsdelivr.net/npm/[email protected]/lib/browser.js"
    );

    self.esbuildService = await self.exports.startService({
      wasmURL: "https://unpkg.com/[email protected]/esbuild.wasm",
    });
  }
}

self.addEventListener('message', async ({ data: { file } }) => {
  await loadEsbuildService();

  const transpiledCode = await self.esbuildService.transform(file.code, {
    target: "es2015",
    loader: "ts",
  });

  self.postMessage({
    code: transpiledCode
  });
});

Note that if you're already running in a worker and don't want startService to create another worker, you can pass startService({ wasmURL, worker: false }). Then it will create a WebAssembly module in the same thread as the thread that calls startService. This means calls to transform will block that thread, but that may be what you want.

[baryla]

The file is in CommonJS format and is intended to be bundled with a bundler. Are you not using a bundler because the file is too big? I can cut the size down to 13kb by minifying it (5kb gzipped).

Gotcha - that makes sense. I could bundle it but there are nice things that come with importing it straight from the CDN.

• I don't have to specify a version and unpkg/jsdelivr will always load the latest version (this is nice if APIs don't change often 😂)
• Keeps Packager super small. I don't want to tightly couple esbuild to Packager because a user may never even invoke a WebWorker with esbuild. For example - if they just want to transpile a Vue file. Packager does allow for plugins so I was actually considering creating an esbuild plugin and then I could bundle esbuild with it.

This is saying that the first four bytes are 3c 21 44 4f which is <!DO. This means you got the URL wrong and the browser is serving you HTML instead of WebAssembly. If you load https://unpkg.com/browse/esbuild-wasm/esbuild.wasm you get a HTML page that links to the correct URL: https://unpkg.com/[email protected]/esbuild.wasm.

You're so right! I can't believe I copied the browse URL form unpkg... I was so eager to try esbuild that I made such an error haha. I can confirm that removing it from the URL made everything magically work :)

Independent of the changes described above, I think this code should work (assuming importScripts can load scripts synchronously):

Yes it totally works. By the time that service.transform is called, the script would've already been imported because it's imported as soon as the WebWorker is created. A small improvement to that code above would be to just move the loadEsbuildService(); call outside of the eventListener.

Note that if you're already running in a worker and don't want startService to create another worker, you can pass startService({ wasmURL, worker: false }). Then it will create a WebAssembly module in the same thread as the thread that calls startService. This means calls to transform will block that thread, but that may be what you want.

Oh! adding worker: false is actually exactly what I want. This is perfect.

One recommendation that I have is to add a doc regarding contributing to esbuild. Personally, I don't know enough Go to be able to contribute to the internal API but TS side I would love to contribute to. I cloned the repo today and I was poking around but I couldn't, for the life of me, figure out how you managed to generate the build that's on NPM for the browser API.

Thank you for help @evanw and once again - amazing work on esbuild. This tool will change (technically already has changed) web development for the better.

[evanw]

I don't have to specify a version and unpkg/jsdelivr will always load the latest version (this is nice if APIs don't change often 😂)

I strongly recommend pinning versions so stuff doesn't break at some point in the future. I do push breaking changes every now and then and I'm relying on other people using versioning to avoid breaking their code.

I just published version 0.5.1 with a new browser.js file. The file is now minified and exports a single esbuild global when it's not imported using require:

<script src="https://unpkg.com/[email protected]/lib/browser.js"></script>
<script>
  (async () => {
    const service = await esbuild.startService({
      wasmURL: 'https://unpkg.com/[email protected]/esbuild.wasm'
    })
    try {
      const ts = 'enum Foo { A, B, C }'
      const { js } = await service.transform(ts, { loader: 'ts' })
      console.log(js)
    } finally {
      service.stop()
    }
  })()
</script>

One recommendation that I have is to add a doc regarding contributing to esbuild. Personally, I don't know enough Go to be able to contribute to the internal API but TS side I would love to contribute to. I cloned the repo today and I was poking around but I couldn't, for the life of me, figure out how you managed to generate the build that's on NPM for the browser API.

I added a small readme to the source directory: https://github.com/evanw/esbuild/tree/master/lib. All build scripts are driven by a single top-level Makefile. For example, the esbuild-wasm package is built using make platform-wasm. I will work more on documentation for contributing.

[baryla]

I strongly recommend pinning versions so stuff doesn't break at some point in the future. I do push breaking changes every now and then and I'm relying on other people using versioning to avoid breaking their code.

Cool that sounds like a good plan.

I added a small readme to the source directory: https://github.com/evanw/esbuild/tree/master/lib. All build scripts are driven by a single top-level Makefile. For example, the esbuild-wasm package is built using make platform-wasm. I will work more on documentation for contributing.

Ah! Makefiles. Thanks for the doc.

I just noticed that the transform API doesn't have the ability to set the format which prevents transpiling CommonJS to ES6+. Is this intentional? This is literally the last piece of the puzzle that I need for esbuild in Packager to be able to handle most of the NPM modules.

[evanw]

I just noticed that the transform API doesn't have the ability to set the format which prevents transpiling CommonJS to ES6+. Is this intentional? This is literally the last piece of the puzzle that I need for esbuild in Packager to be able to handle most of the NPM modules.

The reason is that esbuild doesn't support this yet. This is an active feature request: #109. You can subscribe to that issue for updates.

Keep in mind that while converting ES6 modules to CommonJS is relatively straightforward, converting CommonJS to ES6 modules is error-prone and may not be correct. This is because ES6 imports enforce that the side effects of imported modules take place before the module is evaluated while CommonJS modules can trigger the side effects of imported modules at any time.

The way I'm planning to do the transform is to have each CommonJS module generate a single ES6 default export in the resulting ES6 module for the CommonJS exports object. Each call to require() will become a single ES6 default import.

[baryla]

The reason is that esbuild doesn't support this yet. This is an active feature request: #109. You can subscribe to that issue for updates.

Sorry I must've missed this.

Keep in mind that while converting ES6 modules to CommonJS is relatively straightforward, converting CommonJS to ES6 modules is error-prone and may not be correct. This is because ES6 imports enforce that the side effects of imported modules take place before the module is evaluated while CommonJS modules can trigger the side effects of imported modules at any time.

The way I'm planning to do the transform is to have each CommonJS module generate a single ES6 default export in the resulting ES6 module for the CommonJS exports object. Each call to require() will become a single ES6 default import.

Side effects are indeed a pain and I have experienced just that when trying to support this exact feature. What you are planning does sounds like a good idea and would work great with Packager as it's using Rollup under the hood. I wish I could help with that. Maybe it's time to give Go a proper go ;p