Don't try to describe the Directory Service API in a Thing Description

[benfrancis]

Although in principle I like the idea of describing a Directory using a Thing Description, so far I don't think it's going very well for a few reasons:

1. The desire by some to make it protocol agnostic (e.g. to fit within the limitations of MQTT)
2. The desire by some to make it convenient to consume using the existing (non-normative) WoT Scripting API
3. The need to describe a collection of resources (i.e. the list of Things), for which the Thing Description specification doesn't really have appropriate semantics

Trying to shoehorn the proposed REST API into being described in a Thing Description whilst also being protocol agnostic and convenient for consumption via the Scripting API is creating a bit of a mess in my opinion.

In the interests of being constructive, I'd like to suggest a potential alternative approach which could be more straightforward overall and enable a simple HTTP-only Directory Service API to be defined within the timeframe of the current charter, but be extensible for other protocols if people want to define that in the future.

The idea is that the WoT Discovery specification would extend the Thing Description specification with a new DirectoryAffordance which can be used to point to the API endpoint of a directory. This directory affordance could then have a bunch of additional semantics for describing the API for managing the collection of things, but personally I think it would be simpler for it to just point to the API endpoint and define the expected behaviour as a sub-protocol in the prose of the specification.

An example Thing Description for a directory might look like:

{
  "@context": [
    "http://www.w3.org/ns/td",
    "https://w3c.github.io/wot-discovery/context/discovery-context.jsonld"
  ],
  "@type": "ThingDirectory",
  "title": "My Directory",
  "securityDefinitions": {
    ...
  },
  "security": "combo_sc",
  "base": "https://mydirectory.example.com",
  "directories": {
    "things": {
      "@type": "ThingCollection",
      "title": "My Things",
      "forms": [{
        "href": "/things",
        "subprotocol": "wot-directory"
      }]
    }
  }
}

The API for communicating with a directory could then be defined in the specification, taking a similar approach to that of the Core Profile in the WoT Profile specification. One option could even be to just define the semantics of the DirectoryAffordance in the WoT Discovery specification and then define the Directory API itself as part of the Core Profile.

I recognise this is a big change from the current direction, but I didn't want to criticise the current draft without proposing an alternative.

[benfrancis]

The above example allows for multiple directories, which is a requirement which also just happens to have come up in WebThings (WebThingsIO/gateway#2826), but there could be other ways of fulfilling that particular requirement.

Another couple of options that spring to mind for providing a single directory are a things member of the Thing Description...

{
  "@context": [
    "http://www.w3.org/ns/td",
    "https://w3c.github.io/wot-discovery/context/discovery-context.jsonld"
  ],
  "@type": "ThingDirectory",
  "title": "My Directory",
  "securityDefinitions": {
    ...
  },
  "security": "combo_sc",
  "base": "https://mydirectory.example.com",
  "things": {
    "forms": [{
      "href": "/things",
      "subprotocol": "wot-directory"
    }
  }
}

Or simply a link which links to the top level Directory Service API endpoint....

{
  "@context": [
    "http://www.w3.org/ns/td",
    "https://w3c.github.io/wot-discovery/context/discovery-context.jsonld"
  ],
  "@type": "ThingDirectory",
  "title": "My Directory",
  "securityDefinitions": {
    ...
  },
  "security": "combo_sc",
  "base": "https://mydirectory.example.com",
  "links": {
    "rel": "hosts"
    "href": "/things"
  }
}

[relu91]

I think spawning new ideas and approaches is always good. The proposal seems reasonable and it tries to solve the problem in a creative way but my humble opinion is that we are still far from having something clean/consistent. For example, taking your premises:

1. The desire by some to make it protocol agnostic (e.g. to fit within the limitations of MQTT)
2. The desire by some to make it convenient to consume using the existing (non-normative) WoT Scripting API
3. The need to describe a collection of resources (i.e. the list of Things), for which the Thing Description specification doesn't really have appropriate semantics

If we can't design a TD that is not really protocol-agnostic then we failed the Thing Description spec, is not just a problem of the discovery spec but it's a very profound issue. About the second one, it is not just the Scripting API. If a TD is not really usable from a Scripting API perspective it will very likely not be usable by any client without out-of-band information. Sometimes, it also means that it does not follow the semantic of the WoT Interaction model (which I agree is not really well defined).

Trying to shoehorn the proposed REST API into being described in a Thing Description whilst also being protocol agnostic and convenient for consumption via the Scripting API is creating a bit of a mess in my opinion.

Another thing that I am not sure about. In my mind, since we are creating a brand new service it should be the other way around. we should first create a Thing Description and then map it to a REST API. Then the mapping to other protocols should be a piece of cake.

What I see as the most consistent approach with the current specs is to model the collection of Things as a read-only property and add Actions to query/modify it.

Putting that aside and going back to your proposal, I have one dutb:

"forms": [{
"href": "/things",
"subprotocol": "wot-directory"
}

What would be the default operation here? No operation at all? it just says to the consumers that they are going to use the wot-directory subprotocol for everything right?

[benfrancis]

taking your premises:

To be clear, I'm not saying all of these should be requirements.

1. The desire by some to make it protocol agnostic (e.g. to fit within the limitations of MQTT)

I don't think it's important for the Directory Service API to be protocol agnostic. The reason usually given that Thing Descriptions are protocol agnostic is in order to be able to describe any existing IoT device using any protocol. In this case we are prescribing a new service with a single protocol (HTTP) and there's really no need for it to be protocol agnostic, it's just complicating matters. If someone wants to define a Directory Service API for CoAP or MQTT in a separate or extension specification that's fine.

2. The desire by some to make it convenient to consume using the existing (non-normative) WoT Scripting API

I don't think it's important for the Directory Service API to be easily consumed from the WoT Scripting API. The WoT Scripting API is designed for consuming WoT devices in a protocol agnostic way. The fetch() scripting API is perfectly sufficient for consuming a service over HTTP.

3. The need to describe a collection of resources (i.e. the list of Things), for which the Thing Description specification doesn't really have appropriate semantics

It clearly is important that we be able to describe collections of resources, since that's what a directory is.

If we can't design a TD that is not really protocol-agnostic then we failed the Thing Description spec, is not just a problem of the discovery spec but it's a very profound issue.

I would agree, but Thing Descriptions are designed for describing physical devices, not general HTTP services. The problem in this particular case is not that Thing Descriptions aren't expressive enough, but that they're being used for a use case they're not designed for.

What would be the default operation here? No operation at all? it just says to the consumers that they are going to use the wot-directory subprotocol for everything right?

I agree that isn't clear (and is also a problem for other use cases BTW).

However, I agree this isn't a great use of a Form, so maybe the best way to link to the Directory Service API endpoint is to use a link as in my last example.

    "links": [{
        "rel": "hosts",
        "href": "/things",
        "type": "application/tdd+json"
    }]

This way the Thing Description for a Directory is just used for discovering the directory and locating its top level endpoint. The rest of the API is described in the text of the WoT Discovery specification.

At the simplest level, the application/tdd+json content type can just be defined as a JSON array of Thing Descriptions, and can be parsed as such. A Consumer conforming to the full Directory Service API would also know how to create, update, delete and search that list of Things, by implementing what it says in the specification (as opposed to parsing Forms in a TD).

[benfrancis]

Proposal: #179

[relu91]

Thank you for expending your points @benfrancis ! now I have to admit that you got a point. Having a Thing Description of a TDD would be beneficial to use the same technology stack as if it was just another device. However, it is still an abstract service and if we cannot create a satisfying Thing Description of the API the solution you're describing seems good.

This way the Thing Description for a Directory is just used for discovering the directory and locating its top level endpoint. The rest of the API is described in the text of the WoT Discovery specification.

So in an mDNS scenario, you would fetch the TD to get the concrete endpoint of the TDD plus other relevant metadata (like query types supported, the name of the directory, etc.).

[egekorkan]

I like having a content type for a collection of TDs. One question though, is there a convention on how a content-type be modified by an option to specify this instead of creating a "new" content-type? Like td+json means that it is a special type of JSON, so maybe something like (just throwing a random idea) collection.td+json would mean a collection of TDs serialized as JSON.

[benfrancis]

@egekorkan I see what you're getting at as it may be overkill to define a whole MIME type just to wrap TDs in an array.

Parameter

MIME types can have parameters attached to them in the form type/subtype;parameter=value. One approach could be to distinguish between a Thing and a Directory using a parameter, in a similar way to the proposed type field in DNS-SD discovery used to distinguish between "Thing" and "Directory". e.g.

application/td+json;type=directory

Edit: That is possibly a bit strange because you might expect the above type to represent the TD of a Directory rather than the Directory Service API endpoint itself.

Multipart Type

MIME types can also have multipart types to represent composite documents consisting of multiple parts, each of which can have their own MIME type. I don't think that really helps here though since it would still require a separate MIME type, e.g.

multipart/tdd+json

[egekorkan]

I see, then I would say that the first option, i.e. adding ;type=directory sounds like a good approach

[RoboPhred]

I feel its important that the new solution maintains the ability for things to provide directories. I have been implementing my service as if that was the case, and it elegantly enables a few features I we would not be able to achieve without resorting to carving out nasty exceptions.

The primary use case for this would be the ability to group items together. A common feature in smart home systems, having a flat list of all devices is not particularly useful, and often items are grouped together in some way (locations, rooms, scenes, and so on). Such groups are not simply collections of devices, but also have their own behaviors (turn all group lights on) and properties (set the collective color temperature of all lights). This is in addition to their UX role of providing the user with a filtered list of items within the group.

Another important property directories should support is to be recursive or hierarchical. If the directory was limited to sending a list of things, and things cannot contain directories, then we are now limited to every directory being a flat list. This supersedes the ability to implement groups, and prevents more advanced things that might want to provide multiple sub-behaviors (common in both z-wave with 'endpoints', and in industrial systems where a diesel backup generator is a combination of engine and dynamo).

From an outsiders view and ignorant of the trouble this team has ran into, I feel like having a directory also be a thing seems like a great idea. If DirectoryAffordance capabilities could be placed on Things (and are not limited to a ThingDirectory type that lacks PropertyAffordance and ActionAffordance capabilities), this would work well and solve the use cases I have in mind.

[mmccool]

Propose closing. This is my current understanding:

• The TD is a normative spec
• Anything not covered by the TD is given in prose
  Note that we still have to convert the TD to a TM to be technically correct; a TD example cannot be normative...

[benfrancis]

I'm going to close this issue that I opened, since I think we came to a reasonable compromise on trying to describe a directory using a Thing Description (with the collection of things as a single property). It's not perfect, but it works.