-
-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Scope hoisting for ES6 and CommonJS modules #1135
Conversation
Oh, awesome! I was actually working on something similar. Will take a look in a bit |
src/visitors/hoist.js
Outdated
if (source && specifiers.length > 0) { | ||
if (BREAK_COMMON_JS) { | ||
path.replaceWith( | ||
t.variableDeclaration( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think it should be possible to avoid these extra variables by just renaming the local references to point to the imported one, e.g.
path.scope.rename(specifier.local, IMPORT_NAME_HERE);
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Better! Fixed
src/assets/JSAsset.js
Outdated
@@ -173,7 +174,10 @@ class JSAsset extends Asset { | |||
|
|||
return { | |||
js: code, | |||
map: this.sourceMap | |||
map: this.sourceMap, | |||
'@reflect': { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Rather than adding these to generated
and then removing them later, you could just add them to asset.cacheData
. That already gets written to the cache and you should be able to access it from the packager.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried cacheData
and I didn't get it to work (cacheData.exports
is empty). Should I manually update it? I pushed it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I think you'll have to add it to the asset in the main process from processed.cacheData
here: https://github.com/parcel-bundler/parcel/blob/master/src/Bundler.js#L447
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perfect, fixed
As for commonjs support, I think you can just create an object at the top of the scope like we had before and add all the exports to it, but also leave the individual variables for ES6 usage like you have here. Those objects will get thrown away by the minifier if they aren't used. |
src/packagers/JSConcatPackager.js
Outdated
.split('$' + asset.id + '$expand_exports$' + t.toIdentifier(dep.name)) | ||
.join('$parcel$expand_exports(' + mod.id + ',' + asset.id + ')'); | ||
|
||
if (dep.isES6Import) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We track imports to see if they are from ES6, as well as exports to mark a module as ES6. If they are both ES6, then we can just replace the import names with exports. If a commonjs module is imported with ES6, then the default
import resolves to module.exports
, and other names resolve to module.exports.NAME
.
src/packagers/JSConcatPackager.js
Outdated
return binding; | ||
}) | ||
); | ||
// super.write( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think these re-exports are going to be problematic since they are supposed to be live bindings rather than a copy. Commented for now while we figure it out.
src/visitors/hoist.js
Outdated
// For each specifier, rename the local variables to point to the imported name. | ||
// This will be replaced by the final variable name of the resolved asset in the packager. | ||
for (let specifier of path.node.specifiers) { | ||
if (t.isImportDefaultSpecifier(specifier)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
support more types of specifiers
src/visitors/hoist.js
Outdated
) | ||
); | ||
} else if (t.isImportNamespaceSpecifier(specifier)) { | ||
path.scope.rename( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
importing a namespace e.g. import * from 'blah'
just does the same thing as a commonjs require.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for stuff like this I thought about a Babel transform we could do in the worker, where basically we would replace wildcard imports to named if possible (bailout if the namespace if called). We could also do the same to turn for example const {foo} = require('foo')
to import {foo} from 'foo'
, it may simplify our logic an should be safe enough
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah that could be done. I wonder if minifiers will be able to do that for us though. We're already creating an object with the namespace for every module, so it seems like the minifier should be able to statically resolve those names off the object...
src/visitors/hoist.js
Outdated
|
||
if (!source) { | ||
let declarations; | ||
ExportDefaultDeclaration(path, asset) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added default export support
src/visitors/hoist.js
Outdated
ExportNamedDeclaration(path, asset) { | ||
let {declaration, source, specifiers} = path.node; | ||
|
||
if (source) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Support re-exports, e.g. export {foo} from 'bar'
. Again, this will be problematic I realized, and the code to insert a variable here probably won't work right.
src/visitors/hoist.js
Outdated
} else if (declaration) { | ||
path.replaceWith(declaration); | ||
|
||
if ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Added support for a bunch of different types of declarations. Still more to support, see todos.
src/visitors/hoist.js
Outdated
BINDING: name, | ||
INIT: init | ||
}).declarations[0]; | ||
path.insertAfter( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For each export, we don't create a new variable, we just rename the local variable within the module to the exported name. Also, we add a property on the exports object for use in namespaces and commonjs imports of ES6 modules.
Did a few things, and added some inline comments to describe them. Hope you don't mind me committing to your branch, sorry about that. 😜 Currently the only problematic cases are re-exports:
Currently, we were handling those as variable replacements: we'd make a new variable in the context of the module that is doing the re-exporting, and then dependencies of that module would use the renamed variable instead of the original one. The problem with this is that if the value of the original binding changes, the renamed one will not. So what we really need to do is resolve re-exported names to their original names in the modules that use them. Here's an example that illustrates the problem: a.js export var foo = 2;
export function changeFoo(x) {
foo = x;
} b.js export * from './a.js' c.js import {foo, changeFoo} from './b.js';
console.log(foo) // => 2
changeFoo(10);
console.log(foo) // => 2 (should be 10) |
Thanks for the update, saved me some time 👍 I will look into details later but I noticed it broke tree-shaking and (function () {
/* ASSET: /Users/fathy/Documents/Git/test-parcel/c.js */
var $5$exports = {};
var $5$export$foo = 'hello';
$5$exports.foo = $5$export$foo;
var $5$export$bar = 'tree-shake me if you can';
$5$exports.bar = $5$export$bar;
/* ASSET: /Users/fathy/Documents/Git/test-parcel/b.js */
export * from './c';
/* ASSET: /Users/fathy/Documents/Git/test-parcel/a.js */
console.log('foo', $3$export$foo);
})(); IMHO we should not do About the |
Yeah I commented out the Huh, that's strange about it not removing the unused object. When I tested with uglify on another input, it did seem to remove them. On further testing, it looks like uglify will remove the object but only if there is only one property added to it? Weird. Here's an example. (function () {
/* ASSET: test3.js */
var $3$exports = {};
function $3$export$default() {
return 4;
}
$3$exports.default = $3$export$default;
/* ASSET: index.js */
var $1$exports = {};
function $1$var$add(a, b) {
return a + b;
}
var $1$export$foo = $1$var$add(1, 2);
$1$exports.foo = $1$export$foo;
console.log($1$export$foo);
})(); output: !function(){var o=1+2;console.log(o)}(); |
I pushed a commit with a fix for the wildcard, but it breaks commonjs, I will fix it later. It's messy and I reverted some of your changes, feel free to revert the commit 😄 What I'm trying to do :
Output
import {foo} from './b'
console.log('foo', foo)
export * from './c'
export let foo = 'hello'
export const bar = 'trim me'
export function changeFoo(next) {
foo = next
}
(function () {
/* ASSET: /~/test-parcel/c.js */
var $5$export$foo = 'hello';
var $5$export$bar = 'trim me';
function $5$export$changeFoo(next) {
$5$export$foo = next;
}
/* ASSET: /~/test-parcel/b.js */
/* ASSET: /~/test-parcel/a.js */
console.log('foo', $5$export$foo);
})(); |
Thought it would be useful to summarize all of the possible cases of import/export syntax, and what happens for both ES6 and commonjs modules. Eventually we should have tests for all combinations of these. Imports:
Exports:
|
I changed the way export all is implemented a bit. Rather than making identifiers with the ids of the two modules, I just keep track of whether a dependency was produced by an export all and replace the variables in all subsequent modules pointing to exports from the parent module. Since assets are added to the bundle in order of use, this should be safe. The other thing I did was create the commonjs exports all at once at the end of the file if possible rather than creating an empty object at the start and incrementally assigning to it. This should make it possible for minifiers to prune them more easily. We do incrementally assign to it if the The only problematic one is again export all, which requires an All these string variable replacements do seem somewhat dangerous, but since we're using unique names should be relatively safe. I somewhat like the idea of running babel over everything again in the packager (and we might anyway for minification), but I'm worried about performance. Should probably measure on some larger apps once we get this working fully. |
Actually var _foo = require('foo');
Object.keys(_foo).forEach(function (key) {
if (key === "default" || key === "__esModule") return;
Object.defineProperty(exports, key, {
enumerable: true,
get: function get() {
return _foo[key];
}
});
}); Also apparently |
Turns out Babylon AST can be fully (de)serialised using JSON 😮 const ast = JSON.parse(JSON.stringify(babylon.parse(code))) |
@fathyb I've tested this inside my treeshaking experiment a lil while back and it had circular properties when I've tested it (might have been a non-babel ast that caused the circular props, as I didn't only stringify babel asts). Just wanted to let you know as you might run into the same issue in the future. |
@DeMoorJasper I was a bit afraid of this 😕 thanks for letting me know @devongovett Love the syntax matrices 👍 I tweaked the Babel packager implementation to generate correct exports using the scope (
const {foo} = require('./b')
require('./d')
console.log('foo', foo)
export * from './c'
export const some = 'thing'
export let foo = 'hello'
export const bar = 'trim me'
export function changeFoo(next) {
foo = next
}
console.log('require(./b)', require('./b'))
console.log('require(./c)', require('./c'))
(function () {
/* ASSET: /~/c.js */
var $7$export$foo = 'hello';
var $7$export$bar = 'trim me';
function $7$export$changeFoo(next) {
$7$export$foo = next;
}
/* ASSET: /~/b.js */
var $3$export$some = 'thing';
/* ASSET: /~/d.js */
var $3$exports = {
changeFoo: $7$export$changeFoo,
get foo() {
return $7$export$foo;
},
bar: $7$export$bar,
some: $3$export$some
};
console.log('require(./b)', $3$exports);
var $7$exports = {
changeFoo: $7$export$changeFoo,
get foo() {
return $7$export$foo;
},
bar: $7$export$bar
};
console.log('require(./c)', $7$exports);
/* ASSET: /~/a.js */
var $1$var$_require = $3$exports,
$1$var$foo = $1$var$_require.foo;
var $5$exports = {};
$5$exports;
console.log('foo', $1$var$foo);
})(); |
@fathyb did you commit that? |
RxJS has a file which begins like this : var isArray_1 = require('../util/isArray');
var isArrayLike_1 = require('../util/isArrayLike'); Because of how we do string transformation using var isArray_1 = $352$exports;
var isArrayLike_1 = $352$exportsLike; I replaced the logic in Babel instead with string literals so we don't lose informations like here : import {toIdentifier} from 'babel-types'
toIdentifier('testSass') // testSass
toIdentifier('test-sass') // testSass
toIdentifier('test.sass') // testSass The good news is I successfully compiled RxJS, material-ui, Angular and React! The result on Angular is pretty good. Pushed! |
Yeah I was worried about |
It's getting better! By using Babel's scope we get deep wildcards with tree-shaking without using
import {foo, moduleName, some} from './d'
console.log('foo', foo, moduleName, some)
export * from './c'
export const some = 'thing'
export let foo = 'hello'
export const bar = 'trim me'
export function changeFoo(next) {
foo = next
}
export * from './e'
export * from './b'
exports.moduleName = 'e.js'
(function () {
/* ASSET: /~/c.js */
var $9$export$foo = 'hello';
var $9$export$bar = 'trim me';
function $9$export$changeFoo(next) {
$9$export$foo = next;
}
/* ASSET: /~/b.js */
var $7$export$some = 'thing';
/* ASSET: /~/e.js */
var $7$exports = {};
$7$exports.moduleName = 'e.js';
/* ASSET: /~/d.js */
/* ASSET: /~/a.js */
console.log('foo', $9$export$foo, $7$exports.moduleName, $7$export$some);
})();
!function(){
var o="hello";
var e={
moduleName: "e.js"
};
console.log("foo", o, e.moduleName, "thing")
}();
(function() {
var b = "hello",
e = {};
(e.moduleName = "e.js"), console.log("foo", b, e.moduleName, "thing");
})(); |
f8583ec
to
a2ea1be
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💯🏆
Awesome! 🥇 |
Great work @Fathy 🥇 🏅 |
Haha thanks! But @devongovett did most of the work, so congrats and thanks to him for spending so much time working on this! 🥇 |
).split(''); | ||
|
||
/** | ||
* This is a very specialized mangler designer to mangle only names in the top-level scope. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just curious, why does the mangler fail for topLevel scope? Babel minify supports toplevel mangling and I am interested to know what was the reason to have similar mangling step again here? or is it done for uglify since you are using it as the default minifier?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The point of this mangler is to ONLY mangle the top level scope once all files are concatenation. All other scopes are mangled on a per file level.
* Mangling of names in other scopes happens at a file level inside workers, but we can't | ||
* mangle the top-level scope until scope hoisting is complete in the packager. | ||
*/ | ||
function mangleScope(scope) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can see babel minify license being violated here, Parcel is using the babel minify's mangler code which is licensed(MIT). Can you please mention link to the project and the license?
Closes #1104.
Fixes #392.
eval
import
export * from
)sideEffects
package field (partial support)Output :
a.js
b.js
c.js
--no-minify
--minify