Initial overview of extension structs

[leafpetersen]

This is an initial sketch of an alternative direction for approaching the view/extension type problem. Still in a rough state, putting this up here for some preliminary discussion.

[mit-mit]

I really like the term "struct"; it pairs nicely with the structural identity. I think it's helpful to not call them classes as they are so different from classes, and views seems even more confusing.

[sigmundch]

This connects with my earlier comment: if we view the new primary constructor syntax just as a new syntax available to both classes and structs, then the verbose form is how you'd write such a constructor.

[leafpetersen]

I'm not sure I follow you. My proposal allows you to write an explicit constructor. This comment is about the fact that if I have:

struct Foo(int x = factorial(3));

you get a generated default constructor where the default value for the x parameter is non-const. You currently have no way of writing that explicit constructor:

struct Foo(int x = factorial(3)) {
  Foo([int x]) : // what to write here?
}

[sigmundch]

👍 We definitely want to disallow implementing an interface from a extension struct.

That said, for JSInterop, we are also interested in some form of conformance checking. This is especially useful with mocking. I'll share a few examples here, but I am not sure this is a problem we can solve at this time.

Consider this code:

extension struct Foo(JSObject o) {
  external bool m1(int x);
}

Or the equivalent expansion:

extension struct Foo(JSObject o) {
  bool m1(int x) => js_util.callMethod(o, 'm1', [x]);
}

We'd like to create mocks in Dart that get exported at the JS level, so they can be a valid JSObject to that extension struct (in this case, Foo expects the underlying JSObject to have certain JS properties and methods).

Our current proposal is to say:

class MyFooMock {
  bool m1(int y) => false;
}
...
Foo mock = mockify<MyFooMock, Foo>(MyFooMock());

The type-parameters to mockify give us the magic we need to take MyFooMock and shape it as a JSObject with the expectations of Foo.

We'd like that connection to be more explicit in the syntax, and eventually have a static error if MyFooMock doesn't conform to the extensions declared in Foo. We don't want MyFooMock implements Foo, but if If Foo had an implied interface ExtensionStructInterface<Foo>, we would like to say MyFooMock implements ExtensionStructureInterface<Foo>.

What makes this harder, is that it's not true that Foo and the mock need to match, but how Foo uses the JSObject and the mock do. So to make this example more complicated:

extension struct Foo(JSObject o) {
  @JS('m2')
  external bool m1(int x);

  int m3() => 3;
}

Or the equivalent expansion:

extension struct Foo(JSObject o) {
  bool m1(int x) => js_util.callMethod(o, 'm2', [x]);
  int m3() => 3;
}

The ideal mock class should be:

class MyFooMock {
  bool m2(int x) => false;
}

🤯

/cc @srujzs

[leafpetersen]

Yeah, I don't really see a nice way to get this. One thing you could do with my proposal (almost) is define FooInterface as an abstract class, and say that Foo implements FooInterface, and then you can also have class MyFooMock implements FooInterface . The m1 -> m2 mapping you'd be on your own for... :) but maybe you could mark that in the interface. To get this from my proposal though, you need to make two adjustments (I think)

• You need to allow members from interfaces to be "overridden" in the extension struct (there's nothing problematic with this semantically, but as I discuss in the proposal it has slightly surprising semantics)
• You need to say that JSObject implements all interfaces (or at least interfaces marked in certain ways), since I require that the "on type" implements the interface as well.

[srujzs]

What makes this harder, is that it's not true that Foo and the mock need to match, but how Foo uses the JSObject and the mock do. (etc.)

I mentioned one complication for this earlier on how JS members may have characters in their name that aren't supported in Dart, but maybe it doesn't happen often. Either way, you'd still need a mockify call to wrap it and all that, so it's just slightly cleaner. One reason I wanted to match the Dart names is that the interfacing we describe is actually possible (since MyFooMock.m1 implements Foo.m1, whereas MyFooMock.m2 does not implement Foo.m1, since the language does not care about our renaming annotations).

One thing you could do with my proposal (almost) is define FooInterface as an abstract class...

Yeah, you'd still need to ensure that MyFooMock implements all the members in Foo, unless you start adding weird restrictions that MyFooMock doesn't add any new members.

P.S. I may be reading the proposal wrong, but wouldn't we need JsObject to implement the interfaces in order for an extension struct to implement those interfaces?

It's not that bad if we can't get interfaces for extension structs, it's just a slightly worse user experience for writing mocks.

[leafpetersen]

P.S. I may be reading the proposal wrong, but wouldn't we need JsObject to implement the interfaces in order for an extension struct to implement those interfaces?

@srujzs I'm not entirely sure I understand your question - are you asking about mocking, or in general? In general, yes, this is a question that I wanted to get input from you all. It seems "reasonable" to me, to say that JSObject implements all interfaces for the purposes of this conformance check. So then you can write something like view class MyJSArray(JSObject o) implements List<dynamic>;. It does of course allow you to do all sorts of nasty things: you can pass in any old JSObject to create a MyJSArray, and if you assign it to a List<dynamic> chaos ensues. You can add validation in the constructor if you want (in fact, we could probably even specify this in general), but casts can still get around this.

[srujzs]

Not sure how I missed your second bullet point earlier - you already answered it there, sorry. Having JsObject implement all interfaces is indeed chaos, and I think that would be worse than supporting the interfacing behavior.

It seems like it's fundamentally hard to get this behavior (where an extension struct can provide an interface that is used for another class/struct). Picking your brain here a bit I guess, but why do we want to disallow extension structs defining a signature?

[srujzs]

I'm guessing there was supposed to be more to this comment from the last sentence.

I don't really know what it means for an extension struct to define a proper interface. How does a class implement that interface?

I'd have to imagine it's some synthetic class that's created for that view that can be implemented (like Siggi mentions), but that seems heavy and ugly. Since classes can't implement structs, boxing (with your proposed .struct) doesn't help here either.

Going back a bit here to Siggi's comment:

We don't want MyFooMock implements Foo

Why don't we want this, again? I'd expect the class to conform to every member signature (and if needed, the singular field's signature as well), just like with classes, but I feel like I'm missing something.

I can in principle allow extension structs to implement other extension structs, but I'm not sure that really helps.

Yeah, the lack of state from extension structs (since we don't have fields) would make mocks less useful anyways, so I don't think this is needed.

[leafpetersen]

I'd have to imagine it's some synthetic class that's created for that view that can be implemented (like Siggi mentions), but that seems heavy and ugly.

Sorry, I still don't understand how this works. views use static dispatch. I can perfectly well allow you to say that Foo implements MyStruct, but when I assign a Foo to a MyStruct, the methods that get called will not be the Foo methods, but the MyStruct methods. That's what static dispatch means.

I'm guessing there was supposed to be more to this comment from the last sentence.

Sorry, I meant to delete that last bit. But to close it, we could do something like:

// Magic marker typedef
typedef JSObjectImplements<T> = T;
struct Window(JSObject o) implements JSObjectImplements<List<int>> {}

or really any number of other things. You could go the other way around and do:

typedef JSObjectImplements<T> = JSObject
struct Window(JSObjectImplements<List<int>> o) implements List<int> {}

though this one doesn't generalize well to the n-ary case.

There are other things we could brainstorm.

[sigmundch]

We don't want MyFooMock implements Foo

Why don't we want this, again? I'd expect the class to conform to every member signature (and if needed, the singular field's signature as well), just like with classes, but I feel like I'm missing something.

My point was mainly to make a distinction that static struct/views do not have a virtual interface. I fear that writing MyFooMock implements Foo can lead to confusion because developers will expect an assigment like Foo x = MyFooMock() to work. I'd be more comfortable if Foo's implied interface is distinguished from the type name syntactically somehow. For example, I'd be OK with MyFooMock implement Foo.interface or MyFooMock implements ImpliedInterface<Foo> (that's what I meant by ExtensionStructureInterface<Foo> in the earlier example)

[srujzs]

I can perfectly well allow you to say that Foo implements MyStruct, but when I assign a Foo to a MyStruct, the methods that get called will not be the Foo methods, but the MyStruct methods.

Right, I guess the problem here is assignability as Siggi mentions. I guess implements is the wrong way to think about this, as we only the signature conformance side of that feature, not assignability or subtyping.

And okay, I agree that explicitly mentioning .interface or something similar would avoid the confusion where users think they could subtype.

I do not see any plausible usage for this besides specifically for interop mocking (which we can check anyways, but just at a later stage unfortunately, unless we can maybe leverage the analyzer), so I'm inclined to agree that this is just a small very specific nice-to-have that I'm okay with dropping. After all, why would anyone want to make sure the signatures match if they never plan on using one type for the other? :)

[leafpetersen]

unless we can maybe leverage the analyzer

Possibly a candidate for a lint.

[sigmundch]

One thing that draws me to this syntax is that it aligns with the vision that this is a "restrictive" feature on top of what already exists.

Now I can map that a data class is a class with the following restrictions:

• all of its fields are non-late final.
• it can only extend from an empty data class that acts as a tagging interface
• it can implement interfaces and override methods like regular classes
• unless otherwise specified, the fields by default structurally identify instances of the data class
• it doesn't export a default interface, and hence cannot be "implemented" by other classes in the program

Similarly a static data class (using the proposed static modifier i mentioned earlier) can be explained as a data class that:

• only supports having a single field (aka. the wrapped object)
• always implements a single interface: the type of the wrapped object (aka. the wrapped type)
• doesn't support virtual method dispatch, except for the methods in the wrapped type
• cannot shadow/override methods of the wrapped type, but may statically expose a subset of them.
• has its type available at compile-time only, but later gets erased and replaced by the wrapped type at runtime.

[leafpetersen]

Yeah, I could definitely get behind this story.

[munificent]

One thing that draws me to this syntax is that it aligns with the vision that this is a "restrictive" feature on top of what already exists.

I don't see much value in having a feature that users have to opt in to using and in return... it just prevents them from doing things. I think restrictive features makes sense when in return for fitting within some restrictions, you get other addition for free. If it's just "add this modifier and the compiler will yell at you more", I don't find that very compelling. I understand the general software engineering argument for restrictions making things easier to maintain, but I don't seem those really applying to data classes. A user of a data class shouldn't care whether it was implemented using a data modifier or by hand.

What I do find very compelling is, "Add this modifier and you'll get implementations of ==, hashCode, and copyWith() for free." That's essentially what data does in Kotlin and record in Java and C#. Ideally, any time a user wants those methods with the behavior that the syntactic sugar would produce, then they should be able to use this modifier to get it. The fewer restrictions involved, the more often they'll be able to reach for this modifier and get nicer code.

So, in order to provide implementations of those methods, does the language really need to care if all the fields are final or if the class has an implicit interface? In other words, which of these restrictions are essential? (I mean, obviously, a mutable class with value-based == and hashCode is gonna behave weird if you mutate it. But if the user wants to do that because they know they'll never stuff it in a hash table, why not let them?)

[leafpetersen]

I think you're misunderstanding the usage of "restrictive", which comes from me. What I mean by that is not "I restrict you from doing things!", but rather "as much as possible, these things behave exactly like classes, which you are familiar with, except that there are some things you can't do anymore, in order that we may provide you with these benefits". This is in contrast to some other proposals we've made in this area where "extends" or "implements" meant something fairly substantially different.

If it's just "add this modifier and the compiler will yell at you more", I don't find that very compelling

I don't think there's anything that I'm doing in here where this is the motivation.

The fewer restrictions involved, the more often they'll be able to reach for this modifier and get nicer code.

Agreed. But also, the more benefits they get, they more often they'll reach for this modifier and get nicer code. Essentially all of my choices are (at least intended to be) driven by benefits.

So, in order to provide implementations of those methods, does the language really need to care if all the fields are final or if the class has an implicit interface? In other words, which of these restrictions are essential? (I mean, obviously, a mutable class with value-based == and hashCode is gonna behave weird if you mutate it. But if the user wants to do that because they know they'll never stuff it in a hash table, why not let them?)

It is very much a goal of this to provide/allow structural identity. If you don't agree with that goal, then you won't see that as a benefit (but I do). If you take that as a goal, the semantics get insane to a degree that I'm not comfortable with if you allow mutability.

For the implicit interface, there are at least three aspects that I see as the benefit: one is performance, another is that you get ADTs (sealed families) for free, another is that you get field promotion for free. We could certainly make the ADT thing opt in. I don't love that it requires yet more boilerplate, but I'm not intensely opposed. For field promotion.... maybe we add something to interfaces that requires them to be "stable"? But then there's more leakage.

[leafpetersen]

On further thought couple more points.

On interfaces:

• The sealed family thing is probably wrong, as you point our elsewhere, you can still do this but allow other implementations
• I do care about predictability of representations for the compiler, but we could maybe work with this

So I could be convinced to support a kind of interface here. But again, my goal is to make people convince me to add things to this, rather than the other way around.

On immutability:

• I also do like that the syntax is brief: if I allow both final and non-final, then you need a way of distinguishing, and things get noisier (especially in the default case)
• We could work around the structural identity thing (I think) by saying that if there's a mutable field, identity must be preserved (again, this probably means further restrictions on what kind of interface/subclasses you allow). shrug. I'm open to it, but... again, I'm starting out opinionated. I want the most bang for the buck.

[eernstg]

@sigmundch wrote:

all of its fields are non-late final.

I don't think it is necessary to require non-late, and it is probably quite useful to allow it.

[eernstg]

@leafpetersen wrote:

other proposals we've made in this area where "extends" or "implements"
meant something fairly substantially different

I've compared implements and extends for extension structs and for views in the comment on line 49.

[eernstg]

@leafpetersen wrote:

you get field promotion for free

We can ensure this by enforcing that every concrete struct type is exact in the following sense: If an expression has a concrete struct type S<T1..Tk> and it evaluates to an object o, then the run-time type of o is S<U1..Uk> for some actual type arguments Uj <: Tj (I'm assuming covariance, which is probably very often appropriate for immutable entities).

This is sufficient to ensure that the getter statically known to be induced by a final instance variable will actually be induced by that final instance variable, so the getter is stable, and promotion is safe. Also, getter invocations can be translated into raw memory read operations.

I think this is a very important property of structs, and we should enforce it.

[munificent]

Overall, there is a whole lot of good stuff in this proposal and I'm pretty excited about it.

I left a bunch of comments mainly to braindump stuff that came to mind as I read it so I don't forget before I get back from vacation, but don't feel any need to respond to them.

[munificent]

This is nicer than full classes but still pretty verbose. I need to sit down and really think it through, but I've been mulling over something like:

switch struct Operand {
  case ConstantOperand(Value c);
  case IdentifierOperand(Identifier i);
}

And you could likewise do:

switch class Operand {
  case ConstantOperand {
    ...
  }

  case IdentifierOperand {
    ...
  }
}

The idea is that in a type marked switch, you can have a nested type declaration prefixed with case. That inner type implicitly extends the outer one. (Or implements?) If the inner type is the same kind, we'll let you elide the class/struct/mixin. Or maybe just require it to be the same kind? Can probably nest them to make entire hierarchies.

There's a whole bunch of questions around scoping and whether members and type arguments on the outer type are available on the inner one, but maybe there's something here.

[leafpetersen]

Yeah, this would be really nice.

[rakudrama]

Algebraic types tend to grow to be a discriminated union over many elements. dart2js's SSA HInstruction type has O(100) subclasses in a rich hierarchy. 'pattern matching' is done via a visitor pattern that has base class visitors that delegate to a 'visit' method for the abstract superclass. This is all tedious and error-prone.

This kind of hierarchy is often thousands of lines of code with each element of the hierarchy implementing dozens of methods.

It would be great if the class hierarchy + visitor patterns could be replaced by a hierarchical algebraic (but partially mutable) types.

What would be most disappointing is if a simple hierarchy could be written using something like the proposed syntax above but as the code-base evolves at some point it needs to be completely rewritten in the hierarchy+visitor pattern.

Can multilevel algebraic types with mutable fields be supported?

class Operand switch {
  bool get isSimple;  // abstract member that must be implemented by each 'case'
  case ConstantOperand switch {
    bool get isSimple => true;
    method() { ... }  // methodapplicable to all ConstantOperands
    case StringConstantOperand(String stringValue) {
      List<int> get utf8Bytes => ...;
    }
    case IntConstantOperand(int intValue) {
    }
    ...
  }
  ...
}

[eernstg]

Syntactic nesting is a quite natural and attractive way to describe a tree structure. But it probably doesn't scale very well, it does not support the description of DAGs, and it isn't obvious to me how it would handle parameterization (like type parameters).

I take that to be a hint that we might be quite happy about syntactic nesting as an alternative syntax for certain tree structures. But the name based approach (like our current declarations of classes and their superinterfaces) is a more general tool, and it should always be available.

I also think we can get to a place where minimal examples look really fantastic—but real world code isn't minimal, so we shouldn't over-optimize based on those minimal examples.

[leafpetersen]

Can multilevel algebraic types with mutable fields be supported?

I think there are two separate pieces of your question. I think the patterns/sealed family proposal from @munificent would completely support your use case: you get hierarchies of normal classes, with exhaustiveness checking. The one restriction is that they need to be in the same library.

There's a separate question around a nice short (and possibly hierarchical) syntax for specifying a sealed family. I think @munificent has some ideas around that as well, along the lines of what you sketch out above.

[rakudrama]

What would an inherited copyWith method do that is 'correct'?

[leafpetersen]

Nothing useful. Probably should just disallow it.

[leafpetersen]

I wonder whether this should just be a static method? As long as we keep extension within the library, it works out, and it makes "extends" work out better.

[eernstg]

Commented on several things. The main thing is the comment on line 49, and the invariant in the comment on line 200 starting 'We can ensure this by enforcing'. Finally, I've commented in line 621 on the very idea that the extension struct mechanism should be related to the struct mechanism.

I'd love to have value classes, and the name could very well be struct (because they are more structural than other classes) or something with value.

[eernstg]

Syntactic nesting is a quite natural and attractive way to describe a tree structure. But it probably doesn't scale very well, it does not support the description of DAGs, and it isn't obvious to me how it would handle parameterization (like type parameters).

I take that to be a hint that we might be quite happy about syntactic nesting as an alternative syntax for certain tree structures. But the name based approach (like our current declarations of classes and their superinterfaces) is a more general tool, and it should always be available.

I also think we can get to a place where minimal examples look really fantastic—but real world code isn't minimal, so we shouldn't over-optimize based on those minimal examples.

[eernstg]

It could be an error for a struct interface to have a setter.

[leafpetersen]

Could be. Feels a little overly restrictive to me, with no benefit, but I'll file a discussion issue.

[eernstg]

Right, we would have the following, expressed using views:

view FooBase on SomeSuperType .. {...}
view Foo on SomeType extends FooBase implements Bar {...}

void main() {
  Foo foo = Foo(...);
  Bar bar = foo.box; // `implements` needs boxing.
  FooBase fooBase = foo; // `extends` does not need (nor allow) boxing.

  List<Foo> foos = [foo];
  List<FooBase> fooBases = foos; // OK.
  List<Bar> bars = foos.map((e) => e.box);
}

[eernstg]

I'm not really convinced that extension structs should be structs: The problem is that an extension struct may have an interface which is the entire interface of the underlying representation (the type of the unique instance variable), and then some added members, and there is no reason to assume that this is a proper interface for an immutable entity to have.

If we hadn't had this ability to skip the indirection (so we can do myES.foo() meaning myES.uniqueField.foo()) then the outermost object (existing or not) would be a proper (shallow-)immutable object, and everything would fit nicely. On the other hand, that object wouldn't be able to do anything other than calling the getter uniqueField (plus, presumably the members of Object).

When we conflate the wrapped object with the wrapper (whether or not it exists), we get this conflict.

For instance, if we were to introduce some kind of boxing, I'd assume that the boxed class would simply emerge by taking the extension struct declaration and delete the word extension. We would then have a declaration that specifies a wrapping entity, wrapping a single object. It would be a regular, proper object, and it would have OO dispatch for its methods, and so on.

However, that's a struct, but it would now be a struct that contains forwarding methods to the representation object, including, possibly, some getters and setters which are forwarding to the implicitly induced getters and setters of the mutable instance variables that the representation object has.

This is a weird kind of struct, because it's simply a forwarding wrapper that exposes the entire interface of the wrapped object, including whatever mutability that object offers.

Why, then, would we even have the connection between the struct mechanism and the view mechanism known as extension struct? As far as I can see we could just as well use a normal class when we're boxing an extension struct.

[leafpetersen]

I'm open to splitting out extension structs and structs (e.g. into data class and view class), but I mostly think the mutability argument is a red-herring. Structs aren't deeply immutable, and you can freely add forwarding methods that mutate the underlying fields. Similarly, and extension struct is shallow (not deeply) immutable in the sense that it's not an lvalue, but it does allow delegating mutating members to the unique field. It's true it makes it easier to do so in a way that somewhat obscures the difference though.

Put another way, any interface that can be used in the implements clause of an extension struct can also be used in the interface of a regular struct.

[leafpetersen]

Thanks for all the comments! I added a short section on some possible extensions, including support for more extensive sub-struct relationships. I'll leave this open for another day for any additional comments, and then plan on landing this and filing discussions issues for a number of open design points (including questions raised here).

[leafpetersen]

I've filed #2360 to track this general proposal, with linked sub-issues. Let's continue discussion there and on the sub-issues. Thanks for all the feedback!