finish up

README.md

@@ -21,6 +21,7 @@ Because we are still using the native module loader the edge cases work out comp
 * Live bindings in ES modules
 * Dynamic import expressions (`import('src/' + varname')`)
 * Circular references, with the execption that live bindings are disabled for the first unexecuted circular parent.
+* [Hot reloading extension](#hot-reloading) supporting the `import.meta.hot` API.
 
 > [Built with](https://github.com/guybedford/es-module-shims/blob/main/chompfile.toml) [Chomp](https://chompbuild.com/)
 
@@ -134,31 +135,9 @@ If a static failure is not possible and dynamic import must be used, rather use
 
 When running in polyfill mode, it can be thought of that are effectively two loaders running on the page - the ES Module Shims polyfill loader, and the native loader.
 
-Note that instances are not shared between these loaders for consistency and performance.
+Note that instances are not shared between these loaders for consistency and performance. For this reason it is important to always ensure all modules hit the polyfill path, either by having all graphs use import maps at the top-level, or via `importShim` directly.
 
-As a result, if you have two module graphs - one native and one polyfilled, they will not share the same dependency instance, for example:
-
-```html
-<script type="importmap">
-{
-  "imports": {
-    "dep": "/dep.js"
-  }
-}
-</script>
-<script type="module">
-import '/dep.js';
-</script>
-<script type="module">
-import 'dep';
-</script>
-```
-
-In the above, on browsers without import maps support, the `/dep.js` instance will be loaded natively by the first module, then the second import will fail.
-
-ES Module Shims will pick up on the second import and reexecute `/dep.js`. As a result, `/dep.js` will be executed twice on the page.
-
-For this reason it is important to always ensure all modules hit the polyfill path, either by having all graphs use import maps at the top-level, or via `importShim` directly.
+If instance sharing really is needed, the [`subgraphPassthrough: true` option](#subgraph-passthrough) can be used, although this is not recommended in production since it results in slower network performance.
 
 #### Skip Polyfill
 
@@ -423,6 +402,25 @@ var resolvedUrl = import.meta.resolve('dep', 'https://site.com/another/scope');
 
 Node.js also implements a similar API, although it's in the process of shifting to a synchronous resolver.
 
+### Hot Reloading
+
+Hot reloading support is provided via the separate `dist/hot.js` or `es-module-shims/hot` export.
+
+Load the hot reloading extension before ES Module Shims:
+
+test.html
+```html
+<script src="https://ga.jspm.io/npm:[email protected]/dist/hot.js"></script>
+<script src="https://ga.jspm.io/npm:[email protected]/dist/es-module-shims.js"></script>
+<script type="module-shim" src="/app.js"></script>
+```
+
+While the hot reloading system will work with polyfill mode, it is advisable to use shim mode for hot reloading since the `import.meta.hot` API can only be created for non-native module loads.
+
+The hot reloader will listen to websocket events at `ws://[base]/watch`. Events are strings corresponding to changed file URLs relative to the base URL. The base URL is taken from `document.baseURI`.
+
+`chomp --watch` provides a local server and websocket connection that provides this hot reloading workflow out of the box given the above `test.html` ([Chomp](https://chompbuild.com) can be installed via `npm install -g chomp`).
+
 ## Init Options
 
 Provide a `esmsInitOptions` on the global scope before `es-module-shims` is loaded to configure various aspects of the module loading process:
@@ -439,6 +437,7 @@ Provide a `esmsInitOptions` on the global scope before `es-module-shims` is load
 * [fetch](#fetch-hook)
 * [revokeBlobURLs](#revoke-blob-urls)
 * [mapOverrides](#overriding-import-map-entries)
+* [subgraphPassthrough](#subgraph-passthrough)
 
 ```html
 <script>
@@ -459,6 +458,10 @@ window.esmsInitOptions = {
   enforceIntegrity: true, // default false
   // Permit overrides to import maps
   mapOverrides: true, // default false
+  // Ensure all natively supported subgraphs are loaded through the native loader.
+  // This is disabled by default for performance since the network fetch cache might not be shared
+  // with the native loader. Enabling it avoids some classes of instancing bugs.
+  subgraphPassthrough: true, // default false
 
   // -- Hooks --
   // Module load error
@@ -654,6 +657,48 @@ document.body.appendChild(Object.assign(document.createElement('script'), {
 
 This can be useful for HMR workflows.
 
+### Subgraph Passthrough
+
+_This option is not recommended in production for network performance reasons._
+
+It is a performance optimization in ES Module Shims that even native modules that would support execution in the native loader
+will still be executed as blob URLs through the polyfill loader in order to share the fetch cache used by ESMS itself.
+
+Otherwise, for a module graph like:
+
+main.js
+```js
+import 'dep';
+```
+
+dep.js
+```js
+console.log('dep execution');
+```
+
+`main.js` in the above uses import maps so would be polyfilled in browsers without import maps support. Then if
+the native `import '/dep.js'` were used, the native loader wouldn't have that it in its cache so we could end up
+waiting on a new network request to `/dep.js` even though the polyfill has already fetched it (since in Firefox and Safari, the module network cache and fetch cache aren't always shared). To avoid this ES Module Shims will still use a blob URL for `dep.js`
+providing the source it fetched.
+
+Setting `subgraphPassthrough: true` will cause ES Module Shims to directly inline the dependency as `import '/dep.js'`
+thus causing the native loader to fetch and execute `/dep.js` itself.
+
+This can also be useful to avoid instancing bugs, for example if `dep.js` were imported natively as well:
+
+```html
+<script type="module" src="/dep.js"></script>
+<script type="module" src="/main.js"></script>
+```
+
+In the above without `subgraphPassthrough`, `/dep.js` would be executed natively while `/main.js` would be polyfilled/
+
+When the polyfilled `/main.js` imports `/dep.js` it would be executed through the polyfill loader resulting in the
+`"dep execution"` log output being executed twice.
+
+By setting `subgraphPassthrough: true` this results in a single `"dep execution"` log - the module instance is shared
+between the native loader and the polyfill loader.
+
 ### Hooks
 
 #### Polyfill hook

chompfile.toml

@@ -2,7 +2,7 @@ version = 0.1
 
 extensions = ['[email protected]:footprint', '[email protected]:npm']
 
-default-task = 'test'
+default-task = 'build'
 
 [server]
 port = 8080

package.json

@@ -5,7 +5,8 @@
   "main": "dist/es-module-shims.js",
   "exports": {
     ".": "./dist/es-module-shims.js",
-    "./wasm": "./dist/es-module-shims.wasm.js"
+    "./wasm": "./dist/es-module-shims.wasm.js",
+    "./hot": "./dist/hot.js"
   },
   "types": "index.d.ts",
   "type": "module",

src/env.js

@@ -8,9 +8,18 @@ Object.assign(esmsInitOptions, self.esmsInitOptions || {});
 
 export let shimMode = false;
 
-if (esmsInitOptions.shimMode) setShimMode();
+export let importHook, resolveHook, fetchHook = fetch, metaHook;
+
+if (esmsInitOptions.onimport)
+  importHook = globalHook(esmsInitOptions.onimport);
+if (esmsInitOptions.resolve)
+  resolveHook = globalHook(esmsInitOptions.resolve);
+if (esmsInitOptions.fetch)
+  fetchHook = globalHook(esmsInitOptions.fetch);
+if (esmsInitOptions.meta)
+  metaHook = globalHook(esmsInitOptions.meta);
 
-export let importHook, resolveHook, fetchHook, metaHook;
+if (esmsInitOptions.shimMode) setShimMode();
 
 export const skip = esmsInitOptions.skip ? new RegExp(esmsInitOptions.skip) : null;
 
@@ -27,7 +36,7 @@ if (!nonce) {
 export const onerror = globalHook(esmsInitOptions.onerror || noop);
 export const onpolyfill = esmsInitOptions.onpolyfill ? globalHook(esmsInitOptions.onpolyfill) : () => console.log('%c^^ Module TypeError above is polyfilled and can be ignored ^^', 'font-weight:900;color:#391');
 
-export const { revokeBlobURLs, noLoadEventRetriggers, enforceIntegrity } = esmsInitOptions;
+export const { revokeBlobURLs, noLoadEventRetriggers, enforceIntegrity, subgraphPassthrough } = esmsInitOptions;
 
 function globalHook (name) {
   return typeof name === 'string' ? self[name] : name;
@@ -39,10 +48,6 @@ export const jsonModulesEnabled = enable.includes('json-modules');
 
 export function setShimMode () {
   shimMode = true;
-  importHook = esmsInitOptions.onimport && globalHook(esmsInitOptions.onimport);
-  resolveHook = esmsInitOptions.resolve && globalHook(esmsInitOptions.resolve);
-  fetchHook = esmsInitOptions.fetch ? globalHook(esmsInitOptions.fetch) : fetch;
-  metaHook = esmsInitOptions.meta ? globalHook(esmsInitOptions.meta) : noop;
 }
 
 export const edge = !navigator.userAgentData && !!navigator.userAgent.match(/Edge\/\d+\.\d+/);

src/es-module-shims.js

@@ -19,6 +19,7 @@ import {
   skip,
   revokeBlobURLs,
   noLoadEventRetriggers,
+  subgraphPassthrough,
   cssModulesEnabled,
   jsonModulesEnabled,
   onpolyfill,
@@ -165,7 +166,7 @@ async function topLevelLoad (url, parentUrl, fetchOpts, source, nativelyLoaded,
   if (!shimMode)
     acceptingImportMaps = false;
   await importMapPromise;
-  if (importHook) await importHook(url, fetchOpts, parentUrl);
+  if (importHook) await importHook(url, fetchOpts, parentUrl);  
   // early analysis opt-out - no need to even fetch if we have feature support
   if (!shimMode && baselinePassthrough) {
     // for polyfill case, only dynamic import needs a return value here, and dynamic import will never pass nativelyLoaded
@@ -191,7 +192,7 @@ async function topLevelLoad (url, parentUrl, fetchOpts, source, nativelyLoaded,
       firstPolyfillLoad = false;
     }
   }
-  const module = await dynamicImport(!shimMode && !load.n && nativelyLoaded ? load.u : load.b, { errUrl: load.u });
+  const module = await dynamicImport(!shimMode && !load.n && (subgraphPassthrough || nativelyLoaded) ? load.u : load.b, { errUrl: load.u });
   // if the top-level load is a shell, run its update function
   if (load.s)
     (await dynamicImport(load.s)).u$_(module);
@@ -202,25 +203,20 @@ async function topLevelLoad (url, parentUrl, fetchOpts, source, nativelyLoaded,
 }
 
 function revokeObjectURLs(registryKeys) {
-  let batch = 0;
-  const keysLength = registryKeys.length;
-  const schedule = self.requestIdleCallback ? self.requestIdleCallback : self.requestAnimationFrame;
-  schedule(cleanup);
+  let curIdx = 0;
+  const handler = self.requestIdleCallback || self.requestAnimationFrame;
+  handler(cleanup);
   function cleanup() {
-    const batchStartIndex = batch * 100;
-    if (batchStartIndex > keysLength) return
-    for (const key of registryKeys.slice(batchStartIndex, batchStartIndex + 100)) {
+    for (const key of registryKeys.slice(curIdx, curIdx += 100)) {
       const load = registry[key];
       if (load) URL.revokeObjectURL(load.b);
     }
-    batch++;
-    schedule(cleanup);
+    if (curIdx < registryKeys.length)
+      handler(cleanup);
   }
 }
 
-function urlJsString (url) {
-  return `'${url.replace(/'/g, "\\'")}'`;
-}
+const urlJsString = url => `'${url.replace(/'/g, "\\'")}'`;
 
 let lastLoad;
 function resolveDeps (load, seen) {
@@ -231,6 +227,14 @@ function resolveDeps (load, seen) {
   for (const dep of load.d)
     resolveDeps(dep, seen);
 
+  // use direct native execution when possible
+  // load.n is therefore conservative
+  if (subgraphPassthrough && !shimMode && !load.n) {
+    load.b = lastLoad = load.u;
+    load.S = undefined;
+    return;
+  }
+
   const [imports] = load.a;
 
   // "execution"
@@ -287,7 +291,7 @@ function resolveDeps (load, seen) {
       // import.meta
       else if (dynamicImportIndex === -2) {
         load.m = { url: load.r, resolve: metaResolve };
-        metaHook(load.m, load.u);
+        if (metaHook) metaHook(load.m, load.u);
         pushStringTo(start);
         resolvedSource += `importShim._r[${urlJsString(load.u)}].m`;
         lastIndex = statementEnd;

src/es-module-shims.polyfill.js

@@ -0,0 +1 @@
+import './loader.js';

src/hot-reload.js

@@ -29,13 +29,20 @@ function stripVersion (url) {
   return url.slice(0, -versionMatch[0].length);
 }
 
-const toVersioned = url => url + '?v=' + getHotData(url).v;
+const toVersioned = url => {
+  const { v } = getHotData(url);
+  return url + (v ? '?v=' + v : '');
+}
 
 let defaultResolve;
 
+if (self.importShim)
+  throw new Error('Hot reloading extension must be loaded before es-module-shims.js.');
+
 const esmsInitOptions = self.esmsInitOptions = self.esmsInitOptions || {};
 esmsInitOptions.hot = esmsInitOptions.hot || {};
 Object.assign(esmsInitOptions, {
+  polyfillEnable: true,
   resolve (id, parent, _defaultResolve) {
     if (!defaultResolve)
       defaultResolve = _defaultResolve;
@@ -60,7 +67,7 @@ let curInvalidationInterval;
 
 const getHotData = url => hotRegistry[url] || (hotRegistry[url] = {
   // version
-  v: 1,
+  v: 0,
   // refresh (decline)
   r: false,
   // accept list ([deps, cb] pairs)
@@ -127,6 +134,7 @@ websocket.onmessage = evt => {
   if (data === 'Connected') {
     console.log('Hot Reload ' + data);
   } else {
+    console.log('CHANGE: ' + data);
     invalidate(new URL(data, baseURI).href);
     queueInvalidationInterval();
   }

test/server.mjs

@@ -40,7 +40,7 @@ function setBrowserTimeout () {
     retry += 1;
     if (retry > 1) {
       console.log('No browser requests made to server for 10s, closing.');
-      process.exit(failTimeout || process.env.CI_BROWSER ? 1 : 0);
+      process.exit(failTimeout || 1);
     }
     else {
       console.log('Retrying...');

test/test-preload-case.html

@@ -1,10 +1,10 @@
 <!DOCTYPE html>
 <html>
 <head>
-  <link rel="modulepreload" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/vue.esm.browser.js">
   <script type="importmap">
     { "imports": { "vue": "https://cdn.jsdelivr.net/npm/[email protected]/dist/vue.esm.browser.js" } }
   </script>
+  <link rel="modulepreload" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/vue.esm.browser.js">
   <script type="module" src="/src/es-module-shims.js"></script>
   <script type="module">
     import Vue from 'vue'