Directories should notify subscribers when a TD is updated

[mmccool]

Directories should support a capability to notify a subscriber if a TD changes.
Such changes can include rotation of IDs in order to discourage tracking and enhance privacy (only authorized subscribers would get the new ID).

[farshidtz]

There should also be a mechanism to keep this information for disconnected or late subscribers. For example, MQTT has persistent sessions and retained messages concepts, but relying on a broker adds additional complexity such as the broker access control.

A request-response alternative could be to redirect authorized users from the old resource location to the new one (e.g. HTTP 301 redirect).

Off topic: Need to define a RESTful interaction for changing the ID.

[relu91]

I just want to mention that directories might have also the capability to notify subscribers if something related to one query changes. For example, discovery clients might subscribe not to a set of prefixed events (i.e. TDAdded, TDRemoved), but using a full-blown discovery query like the following:

select ?td where {
      ?td rdf:type wot:Thing;
            rdf:type  sosa:Sensor;
     .....
}

Consequently, the directory might also implement a query-based subscription protocol.

So we could still need this notification mechanism even if we end up with static TDs.

We had some experiments last year. You can find a toy implementation of a Thing Directory capable of handling this notification mechanism here (it is quite outdated but I can revamp a little bit before the next meeting) and, as a reference, it was used as a Thing Directory service inside of our WoT Store implementation (See here).

[zolkis]

IMHO some directories might support TD updates, others may not need to.
But TDs do support versioning - that could be taken as the basis for implementing an update mechanism, optional or not.
If a newer version of a TD is available, there could be an event for that, if the directory declares support for it.
That event could be standardized (for name and content), being supported implicitly, or left to the TD to define it explicitly..

I am leaning towards letting directories be more like applications, described by TDs, as it leaves most room for versatility.
Except when we are coming from security side, they might also be like app stores, imposing constraints, e.g. serving vetted TDs in a secure way to authorized clients - which should be the main focus in standardizing directories IMHO.

[benfrancis]

If a newer version of a TD is available, there could be an event for that

+1. If a directory is described by a TD, there could be an event which is emitted to notify clients when a TD has been updated. This event could have any name, but use a semantic annotation which marks it as a ThingUpdatedEvent.

{
  "events": {
    "deviceUpdated": {
      "title": "Device Updated,
      "@type": "ThingUpdatedEvent",
      "data": {
        "type": "string"
      }
    }
  }
}

[farshidtz]

To summarize, the (optional) directory notification API must allow subscription to three event types for TDs (i.e. created, updated, deleted). Moreover, the API may support server-side filtering of events using SPARQL queries.

What should be inside an event payload?
I think that the event type and TD's ID are sufficient for most applications.
In case of update events, providing the diff may be useful, but this increases the event size and won't scale much if we have to maintain it for disconnected subscribers.

[relu91]

I think that the event type and TD's ID are sufficient for most applications.
In case of update events, providing the diff may be useful, but this increases the event size and won't scale much if we have to maintain it for disconnected subscribers.

I agree the minimum should be eventType and the reference of the update TD (whether it is its ID or something else).

I would not totally discard the diff option. It might be useful to save some network bandwidth. What if we make it an optional field in the event payload? clients might ignore it but who can apply the diff can spare the next API call to get the updated TD. TDDs might also decide not to implement it since it is optional.

[benfrancis]

In #133 we discussed how the current draft combines the created, updated and deleted events into a single event. I propose in that issue that these should be three separate events.

I have also proposed new subscribeallevents and unsubscribeallevents operations in the Thing Description specification to fulfill the requirement of subscribing to all the events in a single operation (w3c/wot-thing-description#1082).

Edit: I should mention that if they are not three separate events, it's not possible for a consumer to subscribe/unsubscribe to/from only a single event if desired.

[farshidtz]

In #133 we discussed how the current draft combines the created, updated and deleted events into a single event. I propose in that issue that these should be three separate events.

I have also proposed new subscribeallevents and unsubscribeallevents operations in the Thing Description specification to fulfill the requirement of subscribing to all the events in a single operation (w3c/wot-thing-description#1082).

Edit: I should mention that if they are not three separate events, it's not possible for a consumer to subscribe/unsubscribe to/from only a single event if desired.

I think adding a fourth subscribe-all endpoint adds complexity without replacing all the current functionality. As mentioned before, the current combination allows subscription to one, multiple (currently two), or all. A new subscribemultipleevents may fit better (which looks like a German word and is hard to read 😃).

I'll summarize my concerns:

• As mentioned in the original PR Add description of the notification API #51, we could separate them and rely on HTTP/2's stream multiplexing to reduce connection overhead. The directories that serve a large number of clients and need to worry about the number of connections will most probably be in the cloud with TLS and HTTP/2 support. The server still needs to maintain the resources for individual handlers, but that is minimal.
• SSE (the protocol used for the events) is designed to support multiple event types in a single stream. The server can set the event type on the stream and clients can easily process by event type (see e.g. https://developer.mozilla.org/en-US/docs/Web/API/EventSource).

Beyond SSE: The subscription control (one/multiple/all) with query arguments also applies to WebSockets. In MQTT, one can either subscribe to multiple using fixed topics events/thingCreated or events/thingDeleted and subscribe to all using a wildcard events/+.

Please see my counterproposal: #133 (comment)

[benfrancis]

I think adding a fourth subscribe-all endpoint adds complexity without replacing all the current functionality. As mentioned before, the current combination allows subscription to one, multiple (currently two), or all. A new subscribemultipleevents may fit better (which looks like a German word and is hard to read smiley).

Maybe this is a matter of taste, but it seems a lot simpler to me than a mess of URI variables. I would be fine with subscribemultipleevents as a convenience operation, but is this limitation actually any different from your proposal? How do you specify multiple (but not all) event types with /events?type=? Do you encode the list of event types as an array in the URL? How do you describe that in a Thing Description?

• As mentioned in the original PR Add description of the notification API #51, we could separate them and rely on HTTP/2's stream multiplexing to reduce connection overhead. The directories that serve a large number of clients and need to worry about the number of connections will most probably be in the cloud with TLS and HTTP/2 support. The server still needs to maintain the resources for individual handlers, but that is minimal.

This seems like an argument for separating them?

• SSE (the protocol used for the events) is designed to support multiple event types in a single stream. The server can set the event type on the stream and clients can easily process by event type (see e.g. https://developer.mozilla.org/en-US/docs/Web/API/EventSource).

This argument could apply equally to both /events (via subscribeallevents) or /events?type (via a single combined event).

Beyond SSE: The subscription control (one/multiple/all) with query arguments also applies to WebSockets. In MQTT, one can either subscribe to multiple using fixed topics events/thingCreated or events/thingDeleted and subscribe to all using a wildcard events/+.

Sure, but WebSockets requires a separate endpoint in its own form because it uses a different protocol scheme. WebSockets also requires defining a sub-protocol since it has no inherent semantics of its own. E.g. in WebThings we use a single WebSocket endpoint per thing in a top level form which supports all operations:

"forms": [
  {
    "op": ["writeproperty", "invokeaction", "subscribeevent"],
    "href": "wss://mywebthingserver.com/things/lamp",
    "subprotocol": "webthing"
  }
]

See w3c/wot-thing-description#1070

But let's try to keep the discussion to the current HTTP-based API with SSE.

[farshidtz]

Maybe this is a matter of taste, but it seems a lot simpler to me than a mess of URI variables. I would be fine with subscribemultipleevents as a convenience operation, but is this limitation actually any different from your proposal? How do you specify multiple (but not all) event types with /events?type=? Do you encode the list of event types as an array in the URL? How do you describe that in a Thing Description?

Passing multiple values to a single query argument is not standardized, but widely accepted as ?value=first&value=second. The servers parse the values into an array.

Some examples:
https://nodejs.org/api/querystring.html#querystring_querystring_parse_str_sep_eq_options.
https://golang.org/pkg/net/url/#URL.Query

Python 3.9.2 (default, Feb 24 2021, 13:30:36) 
>>> from urllib.parse import urlparse, parse_qs
>>> u = urlparse('https://tdd.example.com/events?type=created_td&type=deleted_td')
>>> parse_qs(u.query)
{'type': ['created_td', 'deleted_td']}

This seems like an argument for separating them?

I wasn't trying to argue for the separation. But rather just explaining that connection overheads which would result from decoupling the event endpoints can be solved over HTTP/2. In other words, decoupling does not make it worse for directories serving HTTP/2 clients.

This argument could apply equally to both /events (via subscribeallevents) or /events?type (via a single combined event).

Yes, it also applies to multiple events.

[mmccool]

Resolution: This capability is now included in the specification. There is one subpart which is open but it is linked above: issue #176. So we can close this issue and deal with the last point under that issue.

[egekorkan]

I am not exactly sure how this is fixed in the current specification. How can I subscribe to the thingUpdate type of event for a single TD with a given id? Currently, it seems that the client needs to do filtering on the client-side.

[farshidtz]

@egekorkan please refer to the the Event Filtering section of the working draft which explains how a client can pass one or more TD IDs as query parameters to perform server-side filtering.

Edit:
It looks like this was removed in favour of JSONPath which also got removed in the editor's draft; see #176.

I think it makes sense to add that filtering by ID back.

(Sorry for changing my comment few times. I got confused by looking at different versions of the draft.)

[egekorkan]

I think that the version in the Working Draft is good enough for the id filtering and I would like to see it reintroduced as well :)

[farshidtz]

From Discovery call:

• To discuss whether we should still add feature to the spec, considering the approaching publication date.