[reactivity] Reactivity/Observable/Signals Proposal

[EisenbergEffect]

Reactivity Protocol Proposal

IMPORTANT: Be sure to scroll down in the discussion for a secondary proposal that I think may be a better approach than this one.

The primary purpose of this proposal is to start the discussion on trying to understand whether a general reactivity protocol is feasible, allowing:

• Model/Components systems to decouple themselves from view rendering engines and reactivity libraries.
• View rendering engines to decouple themselves from reactivity libraries.
• Potentially, future native HTML templates and DOM Parts to be able to rely on minimal APIs, even if the browser doesn't yet ship with an implementation.

Achieving this would enable application developers to:

• Swap out their view layers without needing to re-write their models.
• More easily mix multiple view layer technologies together without sync/reliability problems.
• Choose between multiple reactivity implementations, picking the one that has the best performance characteristics based on their application needs. For example, one engine might be faster for initial view rendering, but another might be faster for view updating. Engines could also be selected based on target device. So, a lower memory engine could be used on mobile devices, for example.

A quick implementation of the ideas in this proposal is available here.

Consumers

There are three different consumers of the protocol: reactivity engines, view engines, and model/application developers. Let's look at the proposal from each of these perspectives, in reverse order.

Model and Application Developers

The primary APIs needed by app developers are those that enable them to create reactive values and models. The protocol provides both a declarative and an imperative way of creating property signals. It also provides low-level APIs for creating custom signals.

Example: Declaring a model with an observable property

import { observable } from "@w3c-protocols/reactivity";

export class Counter {
  @observable accessor count = 0;

  increment() {
    this.count++;
  }

  decrement() {
    this.count--;
  }
}

The observable decorator creates an observable property. The underlying protocol doesn't provide an implementation of the signal infrastructure, just a way for the model/app developer to declare something as reactive. We'll look at how the reactivity implementation provides the implementation shortly. There's also an imperative API, which can be used on any object like this:

Example: Using the imperative API to define an observable property

import { Observable } from "@w3c-protocols/reactivity";

Observable.defineProperty(someObject, "someProperty");

Under the hood, both the declarative and the imperative APIs create properties where the getter calls the configured reactivity engine's onAccess() callback and the setter calls the engine's onChange() callback.

The protocol provides a facade to the underlying engine via the Observable.trackAccess() and Observable.trackChange() APIs for consumers that want to create custom signals. Here's how one could create a simple signal on top of the protocol:

Example: Creating a custom signal

import { Observable } from "@w3c-protocols/reactivity";

export function signal(value, name = generateUniqueSignalName()) {
  const getValue = () => {
    Observable.trackAccess(getValue, name);
    return value;
  }

  const setValue = newValue => {
    const oldValue = value;
    value = newValue;
    Observable.trackChange(getValue, name, oldValue, newValue);
  }

  getValue.set = setValue;
  Reflect.defineProperty(getValue, "name", { value: name });

  return getValue;
}

Example: Using a custom signal

const count = signal(0);
console.log('The count is: ' + count());

count.set(3);
console.log('The count is: ' + count());

View Engine Developers

While app developers have a primary use case of creating reactive values, models, and components, view engine developers primarily need to observe these reactive objects, so they can update DOM. The primary APIs being proposed for this are ObjectObserver, PropertyObserver, and ComputedObserver. These are named and their APIs are designed to follow the existing patterns put in place by MutationObserver, ResizeObserver, and IntersectionObserver. A view engine that wants to observe a binding and then update DOM would use the API like this:

Example: A view engine updating the DOM whenever a binding changes

import { ComputedObserver } from "@w3c-protocols/reactivity";

const updateDOM = () => element.innerText = counter.count;
const observer = new ComputedObserver(o => o.observe(updateDOM));
observer.observe(updateDOM);

In fact, you may recognize this as the effect pattern, provided by various libraries, which could generally be implemented on top of the protocol like this:

Example: Implementing an effect helper on top of the protocol

function effect(func: Function) {
  const observer = new ComputedObserver(o => o.observe(func));
  observer.observe(func);
  return observer;
}

Example: Using an effect helper to update the DOM

effect(() => element.innerText = counter.count);

Each of the *Observer classes take a Subcriber in its consturctor, just like the standard MutationObserver, ResizeObserver, and IntersectionObserver. Following the same pattern, they each also have observe(...) and disconnect() methods. The implementation of each of these is provided by the underlying reactivity engine.

Reactivity Engine Developers

A reactivity engine must implement the following interface:

interface ReactivityEngine {
  onAccess(target: object, propertyKey: string | symbol): void;
  onChange(target: object, propertyKey: string | symbol, oldValue: any, newValue: any): void;
  createComputedObserver(subscriber: Subscriber): ComputedObserver;
  createPropertyObserver(subscriber: Subscriber): PropertyObserver;
  createObjectObserver(subscriber: Subscriber): ObjectObserver;
}

The app developer can then plug in the reactivity engine of their choice, with the following code:

Example: Configuring a reactivity engine

import { ReactivityEngine } from "@w3c-protocols/reactivity";

// Install any engine that implements the interface.
ReactivityEngine.install(myFavoriteReactivityEngine);

NOTE: By default, the protocol library provides a noop implementation, so all reactive models will function properly without reactivity enabled.

Here is a brief explanation of the interface methods:

• onAccess(...) - The protocol will call this whenever an observable value is accessed, allowing the underlying implementation to track the access. This is invoked from the getter of a protocol-defined property. Custom signal implementations can also directly invoke this via Observable.trackAccess(...).
• onChange(...) - The protocol will call this whenever an observable value changes, allowing the underlying implementation to respond to the change. This is invoked from the setter of a protocol-defined property. Custom signal implementations can also directly invoke this via Observable.trackChange(...).
• createComputedObserver(...) - The protocol calls this whenever new ComputedObserver() runs so that the implementation can provide its own computed observation mechanism.
• createPropertyObserver(...) - The protocol calls this whenever new PropertyObserver() runs so that the implementation can provide its own property observation mechanism.
• createObjectObserver(...) - The protocol calls this whenever new ObjectObserver() runs so that the implementation can provide its own object observation mechanism.

Since ObjectObserver can be implemented in terms of PropertyObserver and PropertyObserver can be implemented in terms of ComputedObserver, the protocol library provides a FallbackPropertyObserver and FallbackObjectObserver that do just that. This means that the underlying implementation is only required to implement createComputedObserver(). But implementations can choose to optimize property and object observation if they want to by providing observers for these scenarios.

The proposal repo contains a work-in-progress implementation of this proposal. It also contains two test reactivity engine implementations, as well as a test view engine, and a test application.

WARNING: Do not even think about using the test reactivity engines or the test view engine in a real app. They have been deliberately simplified, have known issues, and are not the least bit production-ready. They serve only to validate the protocol.

Open Questions

• Should the protocol enable view engines to mark groups of observers for more efficient observe/disconnect?
  • e.g. Observable.pushScope(), Observable.popScope(), and scope.disconnect().
• Should the protocol provide a way to create observable arrays and array observers?
  • e.g. const a = Observable.array(1,2,3,4,5); and new ArrayObserver(...).observe(a);;
• Should the shared protocol library take on the responsibility of implementing common patterns on top of the protocol such as signal, effect, and resource? (An effect implementation is currently provided as an example.)
• Should the protocol include a standard update queue to ensure timing of subscription delivery or should they be delivered immediately, with the expectation that subscribers handle any sort of batching or DOM update timing?

[EisenbergEffect]

First follow up note: We could probably remove Observable.defineProperty() and the observable decorator from the core and put them into utilities/extensions as long as we surface a standard way of getting/setting property metadata for a target object. That will be needed by component systems. I think a metadata API needs to be there anyway, at least with the approach outlined above.

[Westbrook]

With the parallels to other platform observers, would it make sense to do some history gathering on any proposals to bring similar to actual browsers be of help to the other conversation here?

Might it also be productive to bring some examples of how this approach could replace or modify existing code in libraries that currently provide some of the capabilities referenced above?

[EisenbergEffect]

Alternative Proposal

An alternative to this proposal's onAccess and onChange would be to make a primitive Signal type and combine that with a SignalObserver that functions like the above ComputedObserver allowing a subscriber to run a function and watch any accessed signals. With this model, properties on models and components would just create and get/set signals internally in their getters/setters, which should be easily accomplishable with code similar to what I have above for Observable.defineProperty() and the observable decorator.

I worry that this second approach could be a bit heavy weight since it involves creating an object for every signal. But if this is natively implemented in the runtime, that may not be an issue. It should be easy to polyfill as well.

const count = Signal(0); // from platform
console.log('The count is: ' + count()); // The count is: 0

count.set(3);
console.log('The count is: ' + count()); // The count is: 3

const observer = new SignalObserver(() => console.log('The count is: ' + count()));
observer.observe(count); // observe a signal directly
count.set(4); // The count is: 4

const updateDOM = () => element.innerText = count();
const observer = new SignalObserver(o => o.observe(updateDOM));
observer.observe(updateDOM); // observe signals accessed when invoked

function effect(func: Function) {
  const observer = new SignalObserver(o => o.observe(func)); // from paltform
  observer.observe(func);
  return observer;
}

effect(() => element.innerText = count());

I think most of the original open questions still apply here, such as timing, scopes, arrays, etc.

[EisenbergEffect]

Signal/DOM Integration Proposal

The more I think about having a primitive Signal in JS, the more I like the idea, as it could be integrated more deeply with DOM APIs and yield some interesting opportunities. If DOM APIs accepted both Signal and SignalObserver...

element.setAttribute("some-attr", signal); // automatically updates the attr.
textNode.data = signal; // automatically update the text node
element.className = signal; // updates the classes

A bit wilder...

element.innerHTML = signal; // signal must return TrustedHTML or null
element.appendChild(signal); // signal must return a Node or null

[EisenbergEffect]

Signals as DOM Parts?

At this point, it seems like we've got something that crosses over into the domain of DOM Parts and I wonder if we could handle those situations with:

• A signal implementation in the platform
• Signal integration into key DOM APIs
• A declarative syntax for templates that creates signals and uses them with the DOM APIs.

I realize this is out of scope, but wanted to include it just because I do think these things are related and this could be interesting for DOM Parts. I also think this could work for DSD hydration.

So, imagine a template like this:

<template parsermode="signal">
  Hello {{name}}!
</template>

You can then imagine something like this:

const { fragment, signals } = template.createWithSignals();

signals.name.set("World");
document.appendChild(fragment); // "Hello World!" renders in the document

signals.name.set("W3C"); // "Hello W3C!" renders in the document.

Internally, the browser does something like this:

const name = Signal();
const node1 = document.createTextNode("Hello ");
const node2 = document.createTextNode(name);
const node3 = document.createTextNode("!");
const fragment = document.createDocumentFragment();

fragment.appendChild(node1);
fragment.appendChild(node2);
fragment.appendChild(node3);

return {
  fragment,
  signals: {
    name 
  }
}

[EisenbergEffect]

Signals with DSD

Similar to the DOM Parts for live templates, we could make this work for DSD.

<template shadowrootmode="open" parsermode="prerender">
  Hello {{World:name}}!
</template>

In this case, the browser immediately renders "Hello World!" while also creating a name signal with its initial value set to "World". So, something like this internally:

const name = Signal("World");
const node1 = document.createTextNode("Hello ");
const node2 = document.createTextNode(name);
const node3 = document.createTextNode("!");
const root = host.attachShadow({ mode: "open" });

root.appendChild(node1);
root.appendChild(node2);
root.appendChild(node3);

root.signals = {
  name
};

[EisenbergEffect]

Update:

I met with Daniel Ehrenberg a couple weeks ago and he's going to help me explore adding a signal primitive to the JavaScript language. So, rather than pursuing a protocol right now, we're going to start with work on a standard.

The first steps for me are to meet with various library authors and:

• Understand if such a primitive would be useful to them.
• Understand their use cases.
• Gather feedback on why it should be in the platform.
• Gather technical requirements.

I'll then try to synthesize that information so that a case can be made to tc39 and an initial API proposal can be drafted.

[EisenbergEffect]

Closing this out as we're going to move this to an official proposal repo using the tc39 template. I'll link here once the repo is public.

[Seanmclem]

Closing this out as we're going to move this to an official proposal repo using the tc39 template. I'll link here once the repo is public.

Did you?

[EisenbergEffect]

We have a group established now that is working on putting together the repo and proposal 😄 The repo content is not yet ready yet for public consumption. There's a bit more to write and get worked out. We're making good progress though.

[mattlucock]

A TC39 standardization process will likely take years to complete, assuming it ever will (which is not guaranteed). What are we to do in the meantime?

I think even just building some consensus around the very simple notion of "an object that holds a value and notifies listeners when the value changes" would, today, go a really long way. Indeed, I think such a thing would be flexible enough that you could easily implement more advanced patterns like computed state and reactive side-effects on top of it. Something like this is exactly what we need, and don't have, for the context protocol.

Rough sketch:

class State {
  #value;
  #observers = new Set();

  constructor(value) {
    this.#value = value;
  }

  #callObservers() {
    for (const observer of this.#observers) {
      observer();
    }
  }

  get value() {
    return this.#value;
  }

  set value(newValue) {
    if (newValue !== this.#value) {
      this.#value = newValue;
      this.#callObservers();
    }
  }

  observe(callback, abortSignal) {
    if (abortSignal?.aborted) {
      return;
    }

    this.#observers.add(callback);

    abortSignal?.addEventListener("abort", () => {
      this.#observers.delete(callback);
    })
  }
}

[EisenbergEffect]

I think we are very close to having a v0 signals spec with consensus across almost every major library/framework. This will also come with a JS polyfill implementation. I had hoped the repo would be public by now but there's a desire to make sure "all ducks are in a row" first, and it's taking a bit longer. I'll update this issue when the repo is published.

[mattlucock]

I suppose there's no point trying to also pursue a community protocol if standardization is being pursued. I do note, though, that it does not seem wise to implement the initial version of a TC39 proposal, since you would expect such a proposal to undergo changes as part of the standardization process. I struggle a bit with something like this seemingly being developed behind closed doors, but hopefully those doors won't remain closed for long. I also note that this feels like a generic JavaScript data structures problem and not something that is only relevant to frontend UI.

[Lookwe69]

Any news ?

[EisenbergEffect]

There's been a good bit of work to try to arrive at a v0 spec, with input from virtually every major library/framework. The core group is meeting almost every week now. I had to pull back from the process for a bit myself due to a family medical emergency. However, I have picked back up my involvement in the last week or so and am trying to push forward to getting something public. Realistically I think it's going to be another month or two. The group has a high bar and wants to include a spec-compliant polyfill from the beginning. There are a lot of details to even a v0 proposal. So, it's just taking some time. I remain extremely enthusiastic.

[EisenbergEffect]

Quick update. I think we're going to be able to get something public...very soon. We have a polyfill for the proposal in place now. So, just a few more items to check off. I'll try to put together a blog post to accompany the repo going public so folks who have an interest can get a summary without reading the full spec and all the code if they don't have time (or want to).

[Lookwe69]

https://github.com/proposal-signals/proposal-signals 🚀

[EisenbergEffect]

Yep. That's the repo. We got it public today. 🎊 My blog post is going through final review. Should be out tomorrow. I'll link it here. It will be an easier read than going through the proposal itself.

[EisenbergEffect]

Here's the blog post link:

https://eisenbergeffect.medium.com/a-tc39-proposal-for-signals-f0bedd37a335