Switching to Web IDL

[domenic]

Background

Streams as originally envisioned was meant as a ecosystem-agnostic specification at the level of ECMAScript. As such I wrote it with that specification style (mostly), instead of using Web IDL. See previous discussions in #45 and #732.

Since that time we've seen a shift in the ecosystem, toward using Web IDL more broadly:

• The URL and Encoding specifications (which use Web IDL), among others, are implemented in Node.js. That is, we have significant examples that Web IDL-based specifications can be implemented even in non-web environments.
• The Web Assembly specification settled on using Web IDL, instead of the ECMAScript style, despite being "engine level" and environment-agnostic. This is especially notable because most (all?) implementations of the Web Assembly API are done in the JS engine, instead of in the rendering engine with its IDL binding infrastructure. So this shows that IDL on the specification side is compatible with JS-engine-level implementations in a browser.
• The ECMAScript spec itself is contemplating adopting an IDL, and at least one possible path there is to use an evolution of Web IDL.

Furthermore, not using Web IDL has caused us some pain. I think the pain breaks down in to three main areas:

1. Web platform implementers may be less familiar with this style of specification, which can be frustrating for them (e.g. Exact normative requirements for interface definitions are unclear #732). It can also be harder for them to implement in a way that matches the spec, using the tools available to them; they may need to use JavaScript engine APIs explicitly, or add lots of extended attributes to override Web IDL's default behavior if they use Web IDL anyway.
2. We lose some consistency with the rest of the platform. This category is about minor-ish technical things like over-generic methods (Make pipeThough() non-generic? #961), non-enumerable methods, and especially argument coercion and options bag processing. We gain a little bit of consistency with ECMAScript, but ECMASCript isn't terribly internally consistent anyway, so on balance this is worse.
3. The ecosystem of tooling built up around Web IDL specs is not available to us. This involves things like automatic test coverage via idlharness.js (Missing test coverage for destructuring arguments and GetV. #962), generation of canonical TypeScript definitions (mentioned in JSIDL vs WebIDL #45), and integration with the Web Assembly host bindings proposal. It also causes us problems around how our spec integrates into the IDL ecosystem (Fetch claims to define ReadableStream, but Streams is the actual location fetch#780), and even our spec preprocessor (Figure out what marking up raw JS definitions would need speced/bikeshed#1346). And proposals like get-originals need to have a dedicated branch specifically for streams, instead of just treating these classes like other IDL interfaces.

As such, I think we should reconsider our avoidance of Web IDL. I'd like to use this issue to explore what a switch would look like.

Plan

Concretely, I think there are a few things to consider:

1. Some patterns in the current spec are not expressible in Web IDL. I think this list is exhaustive:
   • Generic methods that don't check their this value: pipeTo() (Make pipeThough() non-generic? #961) and the queuing strategy size() methods
   • Data properties instead of getters: highWaterMark on the queuing strategy classes
   • Non-enumerable methods and properties
   • Async iterators: landing in add @@asyncIterator to ReadableStream #954
   • Missing toStringTag: Question: Should Streams have the prototype for Symbol.toStringTag and toString declared? #952
2. A lot of argument processing in the current spec would change slightly if we switched to idiomatic Web IDL (i.e., using types that weren't any). Especially options bag processing, when converted to dictionaries.
3. A lot of the normative spec text currently assumes it operates on ECMAScript objects, and would need to be carefully switched over. For example, CreateReadableStream's object creation steps would need to move to a vaguer Web IDL style "create an X object". All the IsX() checks would become unnecessary. Etc.
4. Stylistically IDL specs use a different style, e.g. instead of internal slots they use "associated X", and abstract operations are written like "readable stream reader generic cancel steps" instead of "ReadableStreamReaderGenericCancel".

(1) and (2) are the toughest. I would be in favor of just switching wholesale to idiomatic Web IDL, and hoping that there are no web-compat issues. Specifically, moving away from current non-expressible patterns toward more standard ones, and changing argument processing to match how Web IDL dictionaries/optional/etc. work. I think it's likely web-compat will be fine, but who knows, enumerability of methods could throw someone off, or data properties vs. prototypes for the queuing strategies. We could go halfway, e.g. we could add a [NonEnumerable] attribute to Web IDL (it's already in most implementations), or continue defining data properties using ECMAspeak instead of moving highWaterMark to be a prototype getter. Thoughts welcome.

(3) is just a lot of work. It makes me think we may want to do this piecemeal, e.g. we could convert one class at a time.

(4) is silly and I think we should just keep the current style. Switching is not worth it.

Thoughts?

I'd be interested to hear from various audiences. One thing I'm very concerned about is not making implementers feel like we are creating busywork for them. Or that for those implementers like @tschneidereit who implement streams in the JS engine, they now need to uproot all their work and move it to their rendering engine. I'm hopeful that the precedent of Web Assembly alleviates some of these concerns. So I'd appreciate hearing from folks like @youennf / @othermaciej, @tschneidereit / @jorendorff, @ricea / @yhirano, or @travisleithead about whether they'd feel positively or negatively about this sort of shift.

Thanks all, and my apologies for getting us into this situation so many years ago.

[ricea]

I'm in favour, although there are many tricky details. Generally I think the path of least resistance for the standard is likely to also be the path of least resistance for implementations.

An exception is options bag processing, where for implementations it is much easier to use dictionaries, but for the standard would require significant reorganisation. I think on balance the benefit of using dictionaries would be worth it, although there may be some costs I haven't identified.

Nit: size() is a function, not a method. If it was a method that would imply that it was stateful. I think it would be best to just move it off the prototype and onto the object itself.

If I understand correctly, the change would mean that some objects would end up being created in different realms? That stuff is super confusing. Hopefully someone from Firefox can comment on this aspect, as I understand they've been suffering from this.

I agree on (4). Having internal slots look like member variable accesses and operations look like functions has made it easier for Blink's implementation to match the standard. I wouldn't want to move to a more prosey style.

[sideshowbarker]

Whatever changes you end up making, please consider trying to ensure you preserve the existing fragment IDs. Either that or please make sure to give @whatwg/documentation a heads-up if you do end up needing to break them.

The reason I ask that is, MDN docs have already been written for the API, and the MDN articles have links to the existing fragment IDs. So if the fragment IDs change, then all those links break.

In general it’d be nice if editors could keep that consideration in mind when making changes. Maybe it would help to think of it in terms of there being a “docs API” that’s bound to the fragment IDs that exist at the time when the docs are written — so if any fragment IDs change, that’s essentially a breaking change to that “docs API” that should be avoided if possible.

(P.S. I’m really happy about this actual substance of this planned change.)

[chrisdavidmills]

I'm also happy about this; WebIDL is much easier for the MDN writers to figure out in general ;-)

[youennf]

+1 too.
There is some momentum to use streams in various web APIs, which this effort might help.

As an implementor, there might be a minor hit (the WebKit implementation might not be as well aligned as with the current spec wording for instance) but I think it is better to move on anyway.

[tschneidereit]

One thing I'm very concerned about is not making implementers feel like we are creating busywork for them. Or that for those implementers like @tschneidereit who implement streams in the JS engine, they now need to uproot all their work and move it to their rendering engine.

If this actually happens to be the case, I'm very strongly opposed to this change, for what should be obvious reasons. Moving our implementation out of the JS engine is entirely a non-starter, as it would mean starting from scratch.

That said, I do agree WebIDL would otherwise have advantages, so I'll try to figure out a way to keep our implementation while moving to WebIDL. Will update here once I know more.

[Ms2ger]

WebIDL is only a way of specifying the desired behaviour. APIs that are defined in WebIDL can be implemented in a pure JS engine without the help of a WebIDL-based binding generator; it's just more work. That's already been done for https://webassembly.github.io/spec/js-api/ , for example. Claiming that you'd need to "uproot" any work has no factual basis.

[Edited last sentence for tone.]

[tschneidereit]

Ok, after further discussion with @jorendorff, it seems like we should be able to keep our implementation without too much trouble. I'm personally partial to having non-enumerable methods, but it seems reasonably likely that we'll be able to make them enumerable without too much compat trouble, still.

@domenic, one thing that would help is an overview of the behavior changes caused by this. I guess the test changes should uncover most or all of it, so perhaps it's not really needed? Not sure.

[ricea]

@tschneidereit The wpt tests will start using idlharness.js which is even more pedantic than our existing tests and will quickly identify what you need to change to pass.

We've started moving some of Chrome's implementation to WebIDL, so I know some of things that will change.

• The length of argument lists. Currently many arguments are implicitly optional. readable.cancel() means the same as readable.cancel(undefined). In IDL they have to become explicitly optional, which means that ReadableStream.prototype.cancel.length will change from 1 to 0.
• ReadableStream will stringify as "[object ReadableStream]" instead of "[object Object]".

Some things we should change to make it natural as an IDL spec:

• Methods, etc. will become enumerable.
• Property lookups on option bags will become alphabetised, and conversions will be done as specified in the WebIDL spec. For example, instead of the strategy properties being looked up in the order "size", "highWaterMark", they will be looked up in lexicographical order: "highWaterMark", "size". This will probably be the most annoying change for non-WebIDL implementors, but we will have tests for it which will tell you what the order should be.

[annevk]

In private @tschneidereit mentioned a preference for non-enumerable methods. I'm not really familiar with the pros/cons myself, but as we want to start using IDL for more things, including JavaScript objects, it seems like this needs to be configurable. And if it's configurable we need to pick a correct default going forward. (I'm not sure if the way IDL does argument lengths, stringification, and property lookup is controversial.)

I guess those proposing the change to switch streams to IDL don't necessarily want to resolve that question, and I'm not sure it needs to be blocked on it, but I'd like to raise it here so it gets considered.

(See also whatwg/webidl#226 and whatwg/webidl#484 for the corresponding issues on the IDL side, and https://github.com/heycam/webidl/labels/jsidl for the general "jsidl" effort, which hasn't really started, but there's still active interest.)

[domenic]

Glad to hear folks are mostly on-board with this, and that the most contentious issue so far is the non-enumerable vs. enumerable methods ^_^. I don't care much either way there, but I lean toward enumerable since the web has many more enumerable-method classes than JS has non-enumerable-method ones.

I think the next step here is for someone to draft up the changes that would be involved. I'll be co-working with @ricea for a few days from the Tokyo office next month, so things might happen around then, but we'll see; we might just work on transferable streams instead. I'll keep folks posted...

[ricea]

Another difference I noticed:

• For "option bag" types, stream objects are currently permissive about accepting non-objects (see Missing test coverage for destructuring arguments and GetV. #962). For example, new ReadableStream(true, false) currently behaves like new ReadableStream({}, {}) (or you can make it do something else by messing with Boolean.prototype). WebIDL is more strict and only allows undefined, null or objects for dictionary types. Although we could force WebIDL to be permissive using a union, I assume we wouldn't do that.

Re: enumerable vs. non-enumerable, I don't have an opinion except that I'd like to be able to justify whatever choice we make.

[ricea]

One more difference:

• WebIDL doesn't permit a type to have a constructor while not being exposed on the global object. This affects ReadableStreamDefaultReader and WritableStreamDefaultWriter, which are constructible but not exposed.

Possible solutions:

1. Change WebIDL to make the combination of [Constructor] and [NoInterfaceObject] valid.
2. Stop ReadableStreamDefaultReader and WritableStreamDefaultWriter from being constructible.
3. Expose them on the global object.

I prefer option 2., but I suspect I am in a minority.

[domenic]

I tend toward 3. It's the more webby thing to do.

[tschneidereit]

3 seems fine to me, but one thing to note is that if and when we end up adopting WebIDL for JS builtins, we'll need to support non-exposed constructors anyway. ISTM we might as well align streams with that approach.

[ricea]

How do we express [@@asyncIterator] in WebIDL? Are we blocked on whatwg/webidl#580?

[domenic]

We are indeed :(

[ricea]

PR for adding async iterator support to WebIDL: whatwg/webidl#720.