Request for opinion on alternative

[dmchurch]

I see that this proposal has been inactive for some time, but given that it has not yet been withdrawn, I assume there is still at least some interest in the functionality. If it's not too much trouble, I'd like to get some feedback from the champions on an alternative/addition I've written a draft proposal for:

https://es.discourse.group/t/proposal-parser-augmentation-mechanism/2008

This defines (or will define, when complete) a standards-conforming way to describe alternate parser implementations that can be used by ECMA262-compliant engines. The Binary AST importer could be considered the simplest form of such an alternative parser, as its polyfill/validation would consist simply of reading the binary AST and submitting it unchanged to the parser API.

Note that the parser augmentation proposal is not a proposal for runtime-programmable parsing. It doesn't exclude the possibility of engines using the reference implementations as runtime parsers (the polyfill narrative), for example during development, but the primary goal of the proposal is to provide a way to describe native engine functionality, like a Binary AST-enabled browser, or like Deno's inbuilt support for TypeScript (the validation narrative).

If this proposal passes, it would allow the Binary AST to be developed and implemented separately from the TC39 proposal process, much like Flow and TypeScript are. Industry collaborations could define the best form of the AST for optimal performance, and engines could choose to implement them using vendor-defined identifiers, the way that CSS extensions are. Runtimes could use parser feature detection to select and import an AST representation based on which is known to load the most quickly in the running engine, like srcset does for media sources. And, since every alternative parser will be required to define a reference implementation, it would not splinter the ecosystem; rather, it would unify it, since every engine would gain the potential to parse every format.

If that happens, the Binary AST proposal becomes much simpler: rather than asking TC39 to bless a particular representation as the One True AST Of ECMAScript, it will simply be a proposal to standardize the identifiers of a parser implementation already in (hopefully) widespread use.

[kannanvijayan]

Hi @dmchurch, I'm one of the early initiators of the project at Mozilla - although I didn't represent it at W3C.

I like your proposal. It does seem to cleanly generalize a mechanism that would serve a class of proposals, and possibly enable others. I could imagine writing libraries that let you import directly from IDL files and receive fully materialized IPC implementations.

Our immediate goals with BinAST wouldn't be hit with your proposal, though, at least not without a subsequent effort by browsers to internalize our transform and then optimize it. Binary AST's main reason for existence was to eliminate parsing overhead by introducing a low-overhead transform from bytes to an internal AST representation, skipping tokenization and then recursive descent parsing (and backtracking these days). An initial impementation via this mechanism and a BinAST => JS => Parser => AST transform would tick off some boxes, but would slow down the use cases we target: large JS that takes a ton of time to parse. The polyfill adds a transform and doesn't eliminate any existing ones.

BinaryAST only really works in a world where the major browsers implement it as a core syntax and do a direct parse from it to AST, with the polyfill serving as a stopgap for rare cases.

All that said, I think there's a lot of merit in your proposal on its own terms.

[dmchurch]

BinaryAST only really works in a world where the major browsers implement it as a core syntax and do a direct parse from it to AST, with the polyfill serving as a stopgap for rare cases.

This is actually the primary goal behind the Parser Augmentation proposal - it would define a mechanism that allows browsers to specify their own BinAST format.

The key is that the Transformation Description (TD) function is symbolic in nature. It is required to exist, and it represents one way an implementation could transform a given input into an ECMAScript AST, but the host isn't required to execute it as part of parsing - or even, ever. Providing the polyfill would be sufficient for the TD requirement, but then the browser can simply use

syntax "-moz-binary-ast";

at the top of the file as a trigger to switch to the internal, native BinAST implementation.

[dmchurch]

@kannanvijayan I've gotten a much better picture of what a TD will look like in the runtime PA environment. Let's say that you were to make an interface to the native Mozilla binary AST parser using a MozBinAST global object. It has a parse static async method which, when passed an input stream, will parse a binary AST and return it as a JS object representation of the AST - something like the Babylon AST. You can then write the following Transformation Description:

async function parseMozBinAST(parser) {
  const ast = await MozBinAST.parse(parser.inputStream);
  parser.emit(ast);
}
Parser.registerTransformation("-moz-bin-ast", parseMozBinAST);

Now, what a smart PA-compliant engine might do on seeing this registerTransformation call is the following:

1. Prepare a mock parser object, like used for testing. Call the TD function, passing it this mock object.
2. Trap the parser.inputStream access, returning a sentinel value that causes MozBinAST.parse() to return a predictable value. Associate this predictable value with the MozBinAST implementation. (The engine now understands that a request will be made to parse the input stream as a MozBinAST.)
3. Trap the parser.emit() call, recognizing the predictable MozBinAST value (perhaps Symbol.for("-moz-bin-ast")). (The engine now understands that the AST parsed from the input stream is emitted into the document, as-is.)
4. From then on, whenever the parser sees syntax "-moz-binary-ast"; (so, obviously not in this file, which is already parsed, but perhaps in the file that imported this one), or whenever it gets an HTTP header or an import attribute that specifies the "-moz-binary-ast" PA transformation, it will know to hook the input stream up to the MozBinAST parser, and hook the binast parser's output up to the engine's input. No round-trips to JS required, and the TD function is only ever executed once, in a context where the script already had executive control.