Branding guarantees do not match web platform classes, due to return override

[domenic]

Consider the following:

class Base {}
class Ex extends Base {
  #foo = 1;
  isExample() {
    return this.#foo === 1;
  }
}
const o = {};
Ex.__proto__ = function() { return o; };
new Ex;
Ex.prototype.isExample.call(o); // => true

Now consider the analogous setup with the Node and Text classes:

const o = {};
Text.__proto__ = function () { return o; };
new Text();

Text.prototype.splitText.call(o); // throws!!

or, using ES262 classes:

const o = {};
Uint8Array.__proto__ = function () { return o; };
new Uint8Array();

ArrayBuffer.isView(o); // false

This mismatch is worrying. It means that we cannot use private fields for self-hosting any classes involved in inheritance hierarchies.

I'm not sure what a good fix would be, but it seems like we should investigate what can be done here.

[ljharb]

Can you elaborate on why the return override technique is needed to self-host classes involved in inheritance hierarchies?

[jridgewell]

Is this because these classes don't respect overriding the super class?

Text.__proto__ = function () {
  throw new Error('UNREACHABLE');
};
new Text();
// => Text ""

[domenic]

@ljharb it's not; it's only used to illustrate a "failure" of the brand checks in the example.

@jridgewell good catch, I think you are right. I've lost track of the current proposal in that regard: can we still use private fields with constructors that never call super()?

[ljharb]

It means that we cannot use private fields for self-hosting any classes involved in inheritance hierarchies.

@domenic then why does it mean this?

can we still use private fields with constructors that never call super()?

I believe that in a subclass, calling super is a requirement to install any private fields.

[jridgewell]

I've lost track of the current proposal in that regard: can we still use private fields with constructors that never call super()?

Under normal circumstances, no. Fields are installed in a weird step after the parent constructor returns, but before the super returns.

If these classes don't actually have a comparable super() call, then I don't know where they'd be installed.

[domenic]

@ljharb I don't understand; the two things are disconnected.

@jridgewell thanks. That's not great; it drives home the point that we could not use private fields for self-hosting any classes involved in inheritance hierarchies. I guess back to weak maps for those, unless this proposal's champions have any possible fixes?

[nicolo-ribaudo]

Don't weak maps/sets have the same problem?

var set = new WeakSet()
class Base {}
class Ex extends Base {
  constructor() {
    super();
    set.add(this);
  }

  isExample() {
    return set.has(this)
  }
}

var o = {}
Ex.__proto__ = function () { console.log("called"); return o };
new Ex;

Ex.prototype.isExample.call(o); // true

[ljharb]

@domenic i guess i'm not understanding why you'd override __proto__ to self-host in an inheritance chain rather than using class extends and super. (if this is off topic we can certainly discuss separately)

[bakkot]

@domenic:

The behavior you describe looks to me like the branding is on the base class, rather than the subclass. For example, from the behavior you describe, I'd expect Text and Node to look something like

class Node {
  #isNode;
  static assertNode(o) {
    o.#isNode;
  }
}
const assertNode = Node.assertNode;
delete Node.assertNode; // yes, I know this is awkward; decorators can make it a bit simpler, but it's just for illustration

class Text extends Node {
  splitText(offset) {
    assertNode(this);
    // remaining implementation
  }
}

This matches the behavior you observe:

(new Text).splitText(); // works
const o = {};
Text.__proto__ = function () { return o; };
new Text();

Text.prototype.splitText.call(o); // throws!!

Of course, this doesn't match with the observed behavior of not actually invoking the superclass.

I guess back to weak maps for those, unless this proposal's champions have any possible fixes?

Two possible alternatives:

1. make the web platform classes actually invoke their superclass, like a regular class; I imagine implementations would very quickly add a fast path to do what they currently do as long as the superclass is the expected thing
2. make web platform classes have immutable prototype chains (at least up through the base class in the hierarchy), possibly after revisiting Implement meta-object trap(s) to make an object's [[Prototype]] immutable ecma262#538.

[bakkot]

Edit: sorry, I misunderstood. The following works for web platform classes:

const o = {};
Text.__proto__ = function () { return o; };

Text.prototype.splitText.call(new Text) // does not throw

and not for the implementation below. So, yes, I can't see a way of making this work without a WeakMap or changes to how web platform classes work.

Original incorrect idea follows:

---

On further thought, a third alternative would be to do something like:

class Text extends Node {
  #privateStuff = 'abc';
  constructor() {
    if (Text.__proto__ !== Node) {
      return { __proto__: Text.__proto__ };
    }
    super(); // installs #privateStuff on `this`, which is guaranteed to be a Node
  }
  splitText(arg) {
    return this.#privateStuff.split(arg);
  }
}

I think this matches web platform classes in all particulars, at least as described in this thread, down to not invoking artificial (and therefore observable) superclasses:

(new Text).splitText('b'); // works

const o = {};
Text.__proto__ = function () { console.log('reached'); return o; };
new Text(); // does not print 'reached', is not === o

Text.prototype.splitText.call(o); // throws

[jridgewell]

@nicolo-ribaudo: Don't weak maps/sets have the same problem?

Yes, it's a core issue with subclassing.

@ljharb: i guess i'm not understanding why you'd override proto to self-host in an inheritance chain rather than using class extends and super.

He's not. He want's to implement class Text extends Node and Uint8Array natively. But with native class syntax, there's the ability to override the superclass after declaration. These builtin classes do not allow this, meaning there's a difference.

Further, if we were to fix the override difference, there's still the issue of allowing a subclasses privates to be installed on a foreign instance by overriding the superclass.

@bakkot: For example, from the behavior you describe, I'd expect Text and Node to look something like... assertNode(this)

Doesn't this defeat the point of branding by default if I still have to manually brand?

---

@domenic: Do these classes actually use inheritance, other than to define the prototype's inheritance? If they don't, they could be implemented as base classes, then override the prototypes.

[domenic]

To clarify what I meant about "back to weak maps", I think the following would be the only way to implement web platform / ES262 class semantics:

const _data = new WeakMap();
class Text extends Node { // let's ignore CharacterData for this example
  constructor(data = "") {
    const o = Object.create(new.target.prototype);
    _data.set(o, data);
    return o;
  }

  splitText(offset) {
    if (!_data.has(this)) {
      throw new TypeError("Not a Text!");
    }
    return _data.get(this).substring(offset); // (not really what splitText does)
  }
}

It's a shame we couldn't do something similar to

class Text extends Node {
  #data;
  constructor(data = "") {
    const o = Object.create(new.target.prototype);
    o.#data = data; // !?!
    return o;
  }

  splitText(offset) {
    return this.#data.substring(offset);
  }
}

i.e. it's a shame there's no way to use private fields with the return-override feature that apparently is always in use by web platform and ES262 classes. (Does ES402 have any subclasses?)

@bakkot regarding your (1) and (2), do you think it would be reasonable to do either of those two alternatives to all the class hierarchies in the ES262 spec which also have this problem? I think that's mostly the typed arrays, although maybe also the various Function subclasses.

[bakkot]

I personally wouldn't be opposed to changing either 1 or 2 for classes in the spec, but would prefer other options. Spec classes can be weird in other ways anyway.

Per @jridgewell's comment, could you do something like

class Text {
  #data;
  constructor(data = "") {
    this.#data = data;
  }

  splitText(offset) {
    return this.#data.substring(offset);
  }
}
Text.prototype.__proto__ = Node.prototype;
Text.__proto__ = Node;

? i.e. manually wire up the class hierarchy, but keep Text as nominally a "base" class, i.e., such that it does not invoke Node in the process of constructing its instances. This gives you all of

• the correct inheritance for the class objects themselves
• the correct inheritance for instances
• Text has private fields
• constructing Text does not invoke Text.__proto__
• if you modify the Text.__proto__, instances of Text inherit from it, but still have the private fields of Text.

This only works as long as Node does not have its own private fields it needs to install on instances of Text. (Same for your example with Object.create, of course.)

It's a shame we couldn't do something similar to

You can seriously abuse the return-override trick to install private fields on arbitrary objects if you really want to:

class DoNotDoThis {
  constructor(o) {
    return o;
  }
}

class Text extends Node {
  #data;
  constructor(data = "") {
    const thiz = new Node();
    thiz.__proto__ = Text.prototype;
    Text.__proto__ = DoNotDoThis;
    super(thiz);
    Text.__proto__ = Node;
    this.#data = data;
  }
}

buuuut don't.

[domenic]

Spec classes can be weird in other ways anyway.

Do you think they should be weird in these ways, though? I guess I'm unclear why spec classes (in particular ES262 classes) don't follow the same patterns as author-defined ones.

Per @jridgewell's comment, could you do something like

Hmm yeah, that seems superior to my Object.create() approach at least....

This only works as long as Node does not have its own private fields it needs to install on instances of Text.

Right, they kind of do... i.e. these things should be branded as Nodes, not just as Texts.

But I guess this gets into a whole thing where the real requirements are much more complicated than my simple examples so far, e.g. you want "protected" type behavior so the base class can declare private fields that Text's methods can access. And the architecture for doing that properly would probably instead involve a whole different setup (the impl/wrapper distinction).

So not being able to meet the real requirements is OK-ish, I guess. (Which makes me wonder why I opened this thread in the first place...)

I'm not sure there's much actionable left for this thread; thanks for all the attention to detail on walking me through it. I guess it might still be worth pursuing the question as to whether ES262 classes (and probably web classes, following them) should call into their super-constructor.

[jridgewell]

Right, they kind of do... i.e. these things should be branded as Nodes, not just as Texts.

Then my approach won't work. You have to construct a Node instance to install its private fields.

Or, use decorators to extract the necessary private field's key from Node, and another decorator to install that key onto Text. That would let you keep the same "Text is a base class" hack.

This would be simpler if you could install private fields onto foreign objects. It's easy to achieve the same guarantees as branding on top of general encapsulation, but rather cumbersome to do normal property semantics the other way.

[hax]

I guess I'm unclear why spec classes (in particular ES262 classes) don't follow the same patterns as author-defined ones.

This is the real problem.

Branding semantic of current proposal is designed for emulate host object behavior. (at least, one of the reasons)

The question is, why host object have such behavior? Not very clear to me. I guess just the historical reason because it's not easy to make host object fully follow JS object semantics.

On the other hand, do most JS programmers need such branding semantic which just break prototype-inheritance and proxy transparency? Very doubtful.

The most ironic, current fields proposal still can't satisfy simple cases from @domenic , so you still need to use weakmap or other "DoNotDoThis" trick manually!

So I just don't understand what's the rationale of the current design. Sigh...

[littledan]

@domenic Really insightful post.

This issue is a bit broader than just how private interacts with inheritance: If any logic at all is in the superclass constructor, prototype chain mutation can make it not happen, and make other stuff happen instead. Platform objects don't tend to traverse the dynamic prototype chain to run instantiation behavior in their constructor; they just initialize themselves fully from the constructor. So, they're logically following the "original" constructor prototype chain already.

At the limit: If we were to deeply freeze things in built-in modules, then they would already have immutable prototypes, and the issue would not occur. But, this causes other problems as well (e.g., you couldn't monkey-patch behavior onto existing platform-provided objects, only ones that are vended by the wrapper).

Maybe this is the point where we should really reconsider adding a way to freeze the __proto__ of objects. We added "Immutable Prototype Exotic Objects" to explain how Object.prototype, Window.prototype, etc. have an immutable __proto__. New-style classes provided by the system could have the __proto__ of both the constructor and the prototype be frozen.

There's already thought going into these new classes having non-enumerable methods, and putting them in built-in modules, and AFAIK no one complained about freezing Object.prototype.__proto__ or Window.prototype.__proto__, so I don't think the break would be very painful.

For an API to freeze the prototype: What if Object.preventExtensions took an options bag, which you could use to control the level of freezing, like Object.preventExtensions({proto: true, properties: false}). If the options bag is omitted, it's interpreted as today--this means, for example, if a Proxy trap doesn't forward through the options bag, a more conservative option is used, preserving some notions of safety.

Another direction would be to make it so that we could use Reflect.construct on a fixed superclass, and then "manually" add the private fields (and methods) of the current class to the result. I'm not sure what this would look like; probably new syntax.

[kaizhu256]

use static-functions instead of class-methods. problem solved.

[littledan]

@kaizhu256 I believe the fundamental thing here is about how private fields work, not methods.

[domenic]

As an original poster really interested in discussing the issues at hand, I'd strongly request that the champions and repo admins mark off-topic comments as "off-topic" to keep the thread focused, and those wishing to discuss "the design of private fields" and "example UX-workflows" and so on move that to another thread.

[littledan]

@kaizhu256 I've hidden your comment #179 (comment) . There are many other open threads to choose from, or you can start another one, arguing that private is not a good idea. Let's use this thread to work through the issue that @domenic raised.

[rdking]

Funny. This is exactly why I describe branding as merely a means to ensure the desired "fields" exist.

@domenic

It's a shame we couldn't do something similar to...

Why can't you?

class Text extends Node {
  #data;
  constructor(data = "") {
    Text.__proto__ = function() { return Object.create(new.target.prototype); };
    super();
    this.#data = data; // !?!
  }

  splitText(offset) {
    return this.#data.substring(offset);
  }
}

[rdking]

Or better still...

class Text extends Node {
  #data;
  //Using set/getPrototypeOf to avoid __proto__ deprecation
  //Replacing the original value to ensure existence of static members
  constructor(data = "") {
    let original = Object.getPrototypeOf(Text);
    Object.setPrototypeOf(Text, function() { return Object.create(new.target.prototype); });
    super();
    Object.setPrototypeOf(Text, original);
    this.#data = data; // !?!
  }

  splitText(offset) {
    return this.#data.substring(offset);
  }
}

[zenparsing]

One of the original motivating goals of private fields was to "explain JS with JS": to provide syntax for the internal slot capability of built-ins. Issues like this illustrate that private fields are too restrictive to accomplish that goal.

If the answer to this particular issue is to "use WeakMaps", and WeakMaps are the only first-class mechanism we can use to represent non-reflective private state, then I think we should consider whether WeakMaps were the correct feature for self-hosting all along.

I now believe that for the vast majority of application and library code, normal symbols are the most appropriate mechanism, as they interoperate well with all other language features.

Given the above, I'm not sure where private fields (or private symbols for that matter) fits in anymore.

[bakkot]

Private fields are only too restrictive to accomplish that goal because web platform classes and TypedArrays have this behavior where they have both inheritance and mutable prototypes, like ES6 classes, but will always invoke their original prototype even if the prototype has changed, unlike ES6 classes (presumably because this matches the common pre-ES6 "class" pattern using a function). But they can't be called, only new'd, which leaves them in this awkward position between ES5 function-style classes and ES6 class-style classes.

That is, fundamentally this is a mismatch between ES6 classes and web platform classes. If that difference could be patched over - for example, by making classes with this behavior have immutable prototypes - then private fields + ES6 classes would explain web platform classes perfectly modulo cross realm issues. I think that would actually make it easier to understand web platform classes, although it has some ramifications for the ability to polyfill certain kinds of potential future changes.

That said, there's a question of what we mean by "explain". One answer to this issue is to say that it's a perfectly good explanation as long as no one tries to modify the prototype of a built-in or host class, which ought to be vanishingly rare. Frankly I am tempted by that answer.

[hax]

...modulo cross realm issues

@bakkot After all effort, and pay lots of cost, we can never fully emulate host behavior, so maybe we'd better let it go.

I think that would actually make it easier to understand web platform classes, although it has some ramifications for the ability to polyfill certain kinds of potential future changes.

As my 20 years web development experience, it wouldn't. Most programmers just use APIs and never care about branding (they even never heard the word "brand checking"). Only the authors of polyfills and libraries like jsdom care about it, and they have comprehensive knowledge about the quirks of platform classes and don't need use class field to learn the behavior. On the other side, make prototypes immutable could limit the ability of them.

[bakkot]

Most programmers just use APIs and never care about branding

I know many people are not especially inclined to ask how things work. But some people are so inclined, even as beginners. I think it would be a shame not to have a better answer to give to those people than "magic", at least for the broad strokes.

On the other side, make prototypes immutable could limit the ability of them.

Even in the absence of private state, I would still think immutable prototypes (or making these things invoke their superclasses) could be a good change. Having a class-like thing which does not invoke its nominal superclass is weird, from the perspective of ES6 classes. Private state isn't really relevant to that.

[hax]

Actually it's "magic", or let's say it's a host object which can not follow the rules of normal JS objects. I don't think it's hard for beginners to understand it.

And we already know even you can use class field to explain some behavior, you still leave the holes like cross-realm. And you also need to explain why an object have private will fail on proxy, prototype-inheritance while an object without private works fine.

immutable prototypes could be a good change

Actually I'm ok with it. But I don't know whether some author of polyfills would blame you. And you are suggesting a web compatibility breaking change. Good luck.

Private state isn't really relevant to that.

Agree. And I hope we never mixed the very different motivation together, or we will just get the weird thing like current class fields.

[nicolo-ribaudo]

Would immutable prototypes mean that I can't reassign .prototype or that I can't change its properties? In the first case I don't think that it would conflict with polyfills.

[bakkot]

Are you suggesting that we would change these constructors to call there super, given that it would be immutable?

If the constructor chain were immutable, it would not be observable whether these things actually called their superclass or not (because the superclass they start with does not have observable side effects when called). At least, I'd hope that would be the case.

[jridgewell]

So the follow up to that is:

Would immutable supers be available to normal ES6 classes (say, either by default or by some opt-in keyword)?

But even if we do that, it doesn't erase the attack vector, it pushes it earlier. Instead of being able to override the super after the subclass-with-private is defined, I just have to override the constructor before the subclass is defined.

This may not be a huge problem. Local definitions would be fine, and probably module imports? But anything defined on the global object has the potential to be overridden. "X is my super" is a nebulous concept when there's X itself is mutable.

[bakkot]

Would immutable supers be available to normal ES6 classes (say, either by default or by some opt-in keyword)?

See tc39/ecma262#538, but I'd hope so.

it doesn't erase the attack vector, it pushes it earlier

Not entirely sure what you mean by "attack vector" in the context of this thread, although generally speaking it's pretty much impossible to be totally defensive against code which runs before you, except in extremely limited circumstances. Code which is trying to be defensive against other hostile code has to run before that code.

[jridgewell]

Not entirely sure what you mean by "attack vector" in the context of this thread

Using branding as an assertion of "instance of this class". If I can overwrite the super class before you define the subclass, then I can use the same return foreign constructor in the super class and the subclass will treat it as a true instance.

That makes branding really just as "having one implies having all the others" check.

[littledan]

Is there actually a problem with allowing Text's data slot to be installed on a foreign object?

I imagine this would cause significant implementation complexity for browsers, letting all sorts of useless edge cases be visible when they aren't currently. I don't see any reason to go in this direction.

[littledan]

FWIW I documented the "brand" guarantees of this proposal in this FAQ entry. The next step for me is to follow up with the class.instantiate proposal.

[littledan]

I wrote up a very early proposal draft for class.initialize, as described above.

[devsnek]

x-posting from irc:

With the original example that is roughly class X { #y; constructor() { const O = getObjectSomehow(); O.#y = 5; return O; } }.

i think the fact that getObjectSomehow happens to use the prototype is irrelevant. this seems more like a case for something like private symbols. O has no semantic relationship to the instance of X with a private #y that was created.
This is why in the spec, we say ObjectCreate(P, [a bunch of slots]) instead of just ObjectCreate(P).

[littledan]

@devsnek Sorry, I'm not sure what you're getting at. I see this as something that makes private fields analogous to internal slots, which would be meeting the goal expressed in the original post.

[devsnek]

@littledan my understanding was that this was the request:

class X {
  #y;
  constructor() {
    const O = getObjectSomehow();
    O.#y = 5; // this should be possible somehow
    return O;
  }
}

My thoughts, looking at this, is that the class.initialize is the wrong solution for this problem. #y is tied to instances of X, not the result of X.[[Construct]](). There's nothing wrong with overriding the return value and throwing away the instance of X, but if you do that #y is out of the picture.

class.initialize in this case seems like the wrong solution to a more general problem: private state on generic objects. The fact that the generic object is created in the constructor of a class with a private field is beside the point imo. This is probably where private symbols come in if they can achieve consensus.

[Igmat]

@devsnek, yeap Symbol.private will easily handle such case.

[littledan]

@devsnek I don't quite follow. Internal slots are not tied to things that have a certain prototype chain, but rather to what the constructor outputs. Public fields work like this too--they're just a way that the constructor can add things to the object, and don't have anything to do with prototype chains.

The issue here was, even if you mess up the prototype chain of the constructor, you should be able to have the private fields added to the instance. I agree that it's OK to be able to write a class that doesn't have the private fields and instead returns something else. The question was, how should you be able to use the original prototype chain and still add the private fields/methods to the instance, when you want to?

@Igmat I can see how making private symbols that you can add to arbitrary existing instances is a solution to the problem, but does @jridgewell 's proposal (which doesn't do this, at least in the first step) address it?

[rdking]

@littledan @devsnek
I don't know if I'm getting the right impression, but let me ask a question. Since it's already the case that the functions within the lexical scope of the class definition contain the only code that is aware of and able to access private data defined in that class, that all instances of that class contain private data accessible via the same name (the type of that name is irrelevant for this question), and that the only guarantee of brand-checking is that a given object has passed through a given constructor, for the sake of attaching private fields, is it a priority that the object in question is the result of using new on that class?

Or put more concisely, what does it matter where the object came from? Is the constructor of a class that has private fields free to put those fields on any object?

[littledan]

is it a priority that the object in question is the result of using new on that class?

In this issue, we're talking about matching the branding semantics of web platform and JS builtin classes, which do something special with things that were the output of new.

[domenic]

I'm going to unsubscribe from my own thread as this appears to have degenerated into being the same as every other issue thread in this repository, i.e. a place for people to complain about the current proposal and push their counterproposals.

Let me know through out-of-band communication if any more interesting on-topic discussion requires my attention.

[littledan]

I've hidden comments that were about other proposals. Folks can follow up about private symbols in https://github.com/zenparsing/proposal-private-symbols .

[trusktr]

In this issue, we're talking about matching the branding semantics of web platform and JS builtin classes, which do something special with things that were the output of new.

That's a prime example of how existing "weird" (exotic) behaviors are cutting into what a good language feature can actually be, at the expense of end users.

New features don't have to match what existing weird/exotic objects do. Give users the best features for their code, and leave the old mistakes as they are without limiting innovation just to explain existing mistakes.

rant

a place for people to complain about the current proposal and push their counterproposals.

Also a place for people who aren't paid while writing here to feel as if there's little hope. :)

Consider yourself very lucky if you can write here while "at work" and getting paid!

More importantly than creating other proposals, is stopping or slowing this one.

Requiring that some other proposal be accepted in order to stop this one (or else this one will get released and implemented) is not a nice thing for the community to be subjected to.

"Make a better proposal now or else this one will be implemented with its flaws and all."

Its in some ways similar to "Jane gets the electric chair in one week, for killing John, unless you can prove she didn't do it" while having seen someone else do it with your own eyes but not fortunate enough to have evidence.

[littledan]

I think we have examined this topic sufficiently, so I'm closing the thread.