Binding for next PlugFest (Prague 2018)

[mkovatsc]

This examples shall serve as goal definition for the WoT Binding Templates FPWD; let's update this first ("sticky") post to keep it in sync with our consensus. The example is based on Example 1 from the TD spec.

{
  "@context": [ "http://w3c.github.io/wot/w3c-wot-td-context.jsonld" ],
  "@type": [ "Thing" ],
  "name": "MyLampThing",
  "interaction": [
    {
      "@type": [ "Property" ],
      "name": "status",
      "shema": { "dataType": "boolean" },
      "writable": true,
      "observable": true,
      "form": [
        {
          "rel": "readProperty",
          "href": "coaps://mylamp.example.com:5683/status",
          "mediaType": "application/ocf+cbor",
          "coap:methodCode": 1,
          "coap:options": [
            {
              "coap:optionNumber": 2049,
              "coap:optionValue": "1.1.0"
            }
          ]
        }, {
          "rel": "writeProperty",
          "href": "coaps://mylamp.example.com:5683/status",
          "mediaType": "application/json",
          "coap:methodCode": 3
        }, {
          "rel": "observeProperty",
          "href": "coaps://mylamp.example.com:5683/status",
          "mediaType": "application/json",
          "coap:methodCode": 1,
          "coap:options": [
            {
              "coap:optionNumber": 6,
              "coap:optionValue": 0
            }
          ]
        }
      ]
    }, {
      "@type": [ "Action" ],
      "name": "toggle",
      "form": [
        {
          "rel": "invokeAction",
          "href": "https://mylamp.example.com:443/toggle",
          "http:methodName": "POST",
          "http:MessageHeader": [
            {
              "http:fieldName": "Pragma",
              "http:fieldValue": "no-cache"
            }
          ]
        }
      ]
    }, {
      "@type": [ "Event" ],
      "name": "overheating",
      "shema": { "dataType": "string" },
      "form": [
        {
          "rel": "subscribeEvent",
          "href": "mqtt://broker.example.com:8883/MyLampThing/events/overheating",
          "mediaType": "application/json",
          "mqtt:messageType": 3
        }
      ]
    }
  ],
  "link": [
    {
      "rel": "contains",
      "href": "https://mylamp.example.com:443/smart-meter/td"
    }, {
      "rel": "locatedAt",
      "href": "https://myroom.example.com:443/"
    }
  ]
}

[mkovatsc]

If we do not re-use the HTTP vocabulary for CoAP, and hence create a new one, it could be more aligned with the CoAP terms:

• coap:methodCode as unsigned integer instead of http:methodName as string
• coap:optionNumberas unsigned integer instead of http:fieldName as string
• coap:optionValue instead of http:fieldValue

Should we align with CoAP names to be clear or HTTP terms for broader understanding?

[danielpeintner]

I think the example needs to be updated according to w3c/wot-thing-description#68

[mkovatsc]

@mjkoster Did you have time to check the proposal? I will probably not be able to join today's call, I am afraid.

[mjkoster]

Thanks for this example. I agree MQTT should have binary codes as in CoAP. In your example you include only the fields which are used by the protocol driver. We could include optional fields with text strings for human-readability:

"coap:fieldCode": 6
"coap:fiendName": Observe

The link example is illustrative of the difference between link and form. Here links are a peer to interactions

[mkovatsc]

The human-readible part would be recessive I guess, since we need to define how a conflict would be resolved. To avoid this all together, I would leave out these human-readible duplicates for now... Machines can only be forward-compatible when we use the literals that are put into the protocol (e.g., a CoAP client not aware of OCF, but being enabled to communicate by the TD, because it knows how to put the literals into the Options.

@mjkoster How about the rest? Would this sample be the base for the Binding definitions or are there still issues you would like to discuss?

[mkovatsc]

I know that we will be not in sync with the latest proposal in the TD TF, but I understood WoT Binding Templates FPWD should be out about 4 Jan 2018. We can then quickly follow up with another Working Draft relesed together with the new TD Workimg Draft.

[sebastiankb]

Currently, I take some care of the latest TD draft and clean up some bugs in there. I will also take the new inputs of the TD template into account. Currently working in an own branch.

[mkovatsc]

Updated example with the results from call on 19 Dec 2017:

• rel to readProperty, invokeAction, etc.
• Explicit coap vocabulary ("options" instead of "HeaderFields")

[mlagally]

I stumbled over two questions:

1. There is no media type on the "toggle" action - is this intentional?
2. There is no inputData on the "status" property, just a definition of an outputData structure - is this intended?

[mlagally]

why are @type annotations used as arrays?

[mkovatsc]

There is no media type on the "toggle" action - is this intentional?

toggle has no parameters, no payload, and hence no media type.

There is no inputData on the "status" property, just a definition of an outputData structure - is this intended?

Properties in general only have outputData, as inputData must be equal to outputData (REST PUT/GET duality). This is an issue for some platforms, where writes may have an additional parameter (e.g., BACnet priority index or write reason message). It is still unclear, if we should open up to allow different outputData and inputData for Properties.

why are @type annotations used as arrays?

@type allows for arrays and usually there are multiple annotations. We have the convention in TD to use arrays where arrays are possible, even when there is only one entry.

[mjkoster]

Here is a proposal to provide a "schema" keyword to replace "inputData", "outputData", and "value":

{
  "name": "Current brightness",
  "@type": ["Property", "iot:CurrentDimmer"],
  "schema": {
    "type": "object",
    "observable": "true",
    "field": [
      {
        "name": "brightness",
        "@type": ["iot:DimmerData" ],
        "schema": {
          "type": "integer",
          "minimum": 0,
          "maximum": 255
        }
      }
    ]
  },
  "form": []
},

We would propose adding additional terms "inputSchema" and "outputSchema" for the case where they are different, e.g. Actions and Subscriptions

[zolkis]

I like schema, but not inputSchema and outputSchema. A schema definition could be used both in input and output data descriptions, therefore I would separate defining the schema and whether it's used as input or output. In Actions, we could call input data arguments as a sequence of schemas, and call the output data return, specified as a schema. In Events, we could have data, specified as a schema.

In other words, use arguments, return, and eventually data when an in/out context or well-burned-in usage context (like uding data in in events) is associated with the schema, and use schema by default.

[vcharpenay]

Please review my updates on the TD document regarding schema: https://rawgit.com/vcharpenay/wot-thing-description/d9c79993f2bbe92e66af0b7c6fb0aa5852e98358/index.html

[mkovatsc]

Please review my updates on the TD document regarding schema

@vcharpenay Why not as pull request?

• This class embeds JSON Schema declaration
  • should respect @handrews comment min and max vs. minimum and maximum wot-thing-description#84 (comment)
  • Best call it a JSON-LD data schema declaration that builds on the terms defined by [JSON-Schema]
• inputSchema and outputSchema
  • Should take into account Generalize inputData, outputData to "payload" wot-thing-description#66 and DataSchema type names alignments wot-scripting-api#89
  • @mmccool, @zolkis, and I prefer avoiding "input/output"
  • IIRC @mjkoster also mentioned a preference for accept instead of inputSchema; cannot find reference
  • Best candidates appear to be accept for the request payload schema and return for the response payload schema

[sebastiankb]

I will take this latest discussions points here into tomorrow's TD web meeting. Another aspect what we should do is to rename the term form to binding:

• clarifies better the relation to WoT "binding template"
• avoids the confusion of Web Forms as was mentioned here links vs. forms wot-thing-description#85

[mkovatsc]

See #29 (comment) ;)

Maybe you could also discuss the other items of #29

[sebastiankb]

Sorry, I did not check this.

For tomorrow, we have already enough topics, however, I will point to your issue list and ask for comments.

[sebastiankb]

Regarding the renaming of the form term there was the discussion in today's TD web meeting to go for endpoint instead binding. binding can be misleading and endpoint is more related to protocols.

As mentioned in the TD meeting please comment on it until next WoT meeting on next Wednesday (Feb 28th).

[mkovatsc]

Please consider that "endpoint" has meaning in other ecosystems and is something different from what we need. For OCF and IETF, and possibly more, it will create a conflict in meaning!

From the OCF spec (lines 2130ff):

10 Endpoint
10.1 Endpoint definition
The specific definition of an Endpoint depends on the Transport Protocol Suite being used. For the
example of CoAP over UDP over IPv6, the endpoint is identified by an IPv6 address and UDP port
number.

Each OCF Device shall associate with at least one Endpoint [...]

So an "endpoint" is the application that listens at a specific socket address and is usually identified by network address and port number. In our case, an endpoint would be a Servient.

An example for endpoints from the OCF spec:

"eps": [
  {"ep": "coap://[fe80::b1d6]:1111", "pri": 2},
  {"ep": "coaps://[fe80::b1d6]:1122"}, 
  {"ep": "coap+tcp://[2001:db8:a::123]:2222", "pri": 3}
] 

Note that these URIs do not have resource path segments.

I hope everyone can see that "endpoints" are at a different level than Interactions. We are looking for a term that describes the latter. The list the TD provides may contain URIs that are for different endpoints, yes, but the main purpose is to describe how the request needs to be formulated (cf. "rel": "readProperty" ... "writeProperty" ... etc.)

[mkovatsc]

If some want this to be a different concept altogether, where this list identifies endpoints for the Interactions such as [ OCF-endpoint, oneM2M-endpoint, Mozilla-WebThing-endpoint, AzureIoT-endpoint, venderX-endpoint, ... ] and the details about how to formulate the request must be known out-of-band,

then it is probably much better to write this down as full proposal instead of overlapping it with the current concept of completely describing in the TD how to formulate the request. Having two different mindsets defining a single term is doomed to fail.

@zolkis / @mmccool what do you think?

[zolkis]

As mentioned in the call, the context for "endpoint" is given by the hosting class, InteractionPattern. I think that makes quite clear it's a WoT term for endpoint, and still describes it better than form.

Actually we seemed to agree with @mjkoster that the name could be endpoint (instead of form) and the class could be WebLink (instead of Form). @mjkoster explained how WebLink (with target attributes) could work both with OCF and WoT (and others), being quite general concept. Of course, we could also call it then weblink (perhaps just link would be not specific enough) instead of form.

In the OCF JS APIs we ended up defining a Destination, but I don't think that name is very fortunate. I could imagine working with endpoint or weblink for a while.

[mkovatsc]

So the mismatch between OCF usage and WoT TD usage is fine with you and the risk of confusion is limited?

And also it is not the case that you are asuming a different concept as outlined above?

[zolkis]

So the mismatch between OCF usage and WoT TD usage is fine with you and the risk of confusion is limited?

I would think so. However, I cannot vouch for others. If you think it's confusing, then I suggest using the name weblink with type WebLink as defined in RFC-8288. That should be pretty clear, but in the spec text we should say we use this as an endpoint (currently the text does say it).

[zolkis]

Oh, I see there has been a similar suggestion in w3c/wot-thing-description#94 (comment).

[mjkoster]

We seem to mostly agree that we should look for a better term than "form".

"binding"
In the discussion we pointed out that "binding" includes the dataSchema and the element most recently known as "form".

"endpoint"
The commonality between OCF "ep" and this element is that it offers alternate locations for the same information. Our use is per interaction and includes path. OCF is per device and consists of scheme and authority.

Some other choices people have made for similarly-purposed elements:

Hydra:
"operation" on rdf:property, supported by a resource, has: method, expects, returns, statusCode
uses "link" to point to property instances

ALPS:
HTML vocabulary: "form" element, has "action" which contains URI, "method" for GET, POST, etc. and "input" to describe the submission semantics

JSON Hyper-schema:
"submissionSchema", submission link, http oriented with headerSchema, etc.

HAL:
Only links

CORAL:
"form" with "method", "rel", "href", "form-data", "form-fields"

HSML:
"form" with "method", "rel", "href", schema embedding using data templates

[mjkoster]

I would describe the element in question as fundamentally a typed link that contains additional submission semantics, including transfer/transport layer bindings and settings.

In that sense it has a lot in common with JSON Hyper-schema. But JSON Hyper-Schema is a schema language to constrain link construction, though it has been supposedly somehow repurposed as a hypermedia control

We also have some idea to process return codes, so borrowing from Hydra may be useful; however Hydra is a different style RDF integration with some potential conflicts with TD.

Being a typed submission link, I am back to thinking we should still consider "link" and use the "rel" value to categorize it as one possible option. This means we can have links anywhere, which are interpreted according to their "rel" values.

[handrews]

@mjkoster

JSON Hyper-schema:
"submissionSchema", submission link, http oriented with headerSchema, etc.

headerSchema is not intended to be HTTP-oriented. It and targetHints are intended to describe protocol-specific metadata, with applications deciding how to use these fields based on the URI schema in the resolved href.

So a link resolving to an https:// URI would look at these fields expecting HTTP headers, while a coaps:// link could look for HTTP headers and also information about acceptable values for the CoAP header (I think- I just skimmed the CoAP RFC for like 30 seconds apologies if this makes no sense).

If anyone wants to provide examples of other protocols / URI schemes, I can speak to how these fields would be used with them.

We've considered changing headerSchema to metaDataSchema to get people to not associate it as strongly with HTTP headers (there's only one implementation underway so far, two if I can carve out some time, so changing the name won't be too disruptive yet).

[handrews]

In that sense it has a lot in common with JSON Hyper-schema. But JSON Hyper-Schema is a schema language to constrain link construction, though it has been supposedly somehow repurposed as a hypermedia control

@mjkoster : As I'm the primary author of the last two Hyper-Schema specs, I'm a bit curious about this "supposedly" comment. What part is unclear? JSON Schema as a whole offers both assertions (which are used to implement validation) and annotations (of which links is a complex example that can depend on the instance value- title and description are simple examples that are values independent of the instance)

While assertions function as a constraint system, I don't see annotations in general, or links in particular, as constraints, so I'm also curious about that viewpoint.

[handrews]

@mjkoster

I would describe the element in question as fundamentally a typed link that contains additional submission semantics, including transfer/transport layer bindings and settings.

I feel this description fits JSON Hyper-Schema. I don't think that automatically means that JSON Hyper-Schema is ideal for your purposes here, but I don't think this description excludes it.

Hyper-Schema links in draft-07 attempt have a structure independent of transfer/transport layer bindings, but also have fields (targetHints and headerSchema) that allow specification of such things without tying Hyper-Schema itself to any particular transfer/transport.

For form equivalents, there are three options:

• hrefSchema defines how to submit input to resolve a URI Template. This can be used to implement HTML GET forms, although it is more flexible than that as it is not limited to the query string or to application/x-www-form-urlencoded, and the URI construction can be used with any protocol or protocol method. You could use this to dynamically construct URNs, even.

• targetSchema defines the target representation, so it can be used as a form for PUT or analogous operations. Together with a media type such as application/merge-patch+json, it can be used to describe a form for a PATCH.

• submissionSchema defines non-representation data that can be processed by the target resource. In HTTP and CoAP this is implemented by POST, but any operation of any protocol that takes non-representation data (or representation of a related resource, such a creating via collection) can be described with submissionSchema.

Obviously hrefSchema can be used alongside either of the other two. The other two can both be defined on the same link but will only be used one at a time. You are either sending representation data or non-representation data in a payload, which determines which schema to use.

[mjkoster]

Hi Henry,

I included the word "supposedly" because I don't know where to find examples of json hyperschema syntax being used for instances of hypermedia controls. The LDO seems like more of a description of how to construct instances of links, rather than a serialization for the links themselves. Does my question make sense?

In other words, can you point me to some examples showing how resource instances can use json hyperschema as a serialization format that the the client fetches to use as a hypermedia control? Or otherwise explain what misunderstanding I have about how json hyperschema is used?

Is it intended to use the hyperschema keywords in link serialization?

Or is it that we would need to use json hyperschema to describe links in the payloads (in which case the link serializations could follow RFC6690 or core-links-json) ?

What does it mean to "use" json hyperschema?

Does it help to consider the use case for json schema as a starting place?

[handrews]

@mjkoster thanks for the explanation, that makes sense! I'll try to explain (and maybe this will help come up with better wording for the next draft).

The LDO is definitely a description of how to construct instances of links. As of draft-07, the specification also contains a recommended* output format that could be used to serialize a set of actual links based on applying a hyper-schema to an instance.

There are two primary way of using Hyper-Schema:

Client Side

On the client side, a hyper-schema implementation can be used to construct links as needed. This reduces the bits on the wire, and distributes the cost of link construction from the server (where it needs to be done unconditionally) to the client (which can opt to only construct links that are requested).

The client-side implementation would presumably produce the recommended format, although it could instead produce something like an HTTP link header (RFC 8288 serialization).

This approach is favored by people with servers with constrained resources. Link representation and construction is offloaded from the server entirely- the server need only identify the relevant hyper-schema with a URI.

Since hyper-schemas could be independently distributed (or compiled in) to clients, this could also remove network activity needed to fetch the hyper-schemas (although this gets tricky as APIs evolve and you need to distribute new hyper-schemas- both dynamic fetching and pre-distribution have their strong and weak points).

Client side also allows for progressive enhancement / graceful degradation when there is resistance to hypermedia, or a legacy API to which hypermedia needs to be added unobtrusively.

Client side is also the only option that reliably supports forms well, as you need to have the various LDO schemas available on the client side to render and/or validate the input controls. Most JSON hypermedia formats such as HAL do not support forms (I think Siren does, and it's been a while since I looked at Hydra but I recall it being one of the more complete systems).

Server Side

On the server side, hyper-schema could be used to construct one or more other serializations of links. HTTP or CoAP link headers would be one option. Another would be to construct formats such as JSON HAL in response to content negotiation. If the content negotiation requests application/json, then the server can still include a link to the hyper-schema, thereby supporting both server-side and client-side usage.

We haven't put much thought into constructing the hyper-schema recommended output format on the server side and sending it as its own media type, or anything like that. A few people have been really interested in the server-side option recently, but historically the client side use case has been the focus.

---

*the output format is "recommended" because there wasn't consensus on whether it was better to make the format required or allow implementations to use whatever output makes sense for them- hopefully we'll get feedback on this.

[handrews]

@mjkoster I missed part of your comment initially, here's a response to the rest of it, some of which overlaps with the last comment:

Is it intended to use the hyperschema keywords in link serialization?

Not directly, although I suppose you could do that with the recommended output format. It seems like in-line hypermedia formats are fairly well covered by other media types and headers, though.

Or is it that we would need to use json hyperschema to describe links in the payloads (in which case the link serializations could follow RFC6690 or core-links-json) ?

Yes, this seems closer to Hyper-Schema's intended usage- you could use it on either the client or server side to produce both of these formats. Whether that's useful or just an irrelevant extra step, I'd have to understand a bit more about the flow of things. I haven't been following this project as much as I would have liked, I'm afraid.

Anyway, hyper-schema could allow you to generate these documents from an instance (or if there is no instance-dependent data, then it could serve as an alternate serialization that could be translated with or without an instance). The additional link fields (e.g. rt, ct) could be handled as extension keywords in the LDO.

For draft-08 we are working on making it easier to declare vocabularies in a modular fashion, so you could say "this document uses the standard Hyper-Schema vocabulary and the CoRE extension vocabulary" in a machine-recognizable way. Currently this isn't possible- either you declare the standard meta-schema (and have no way to recognize extensions), or you declare an extended meta-schema (and have no way for implementations that don't recognize the extension to recognize the underlying standard vocabulary). This clearly needs to be fixed.

For forms, you need user input as well as the hyper-schema and (possibly) instance data to construct the form request.

What does it mean to "use" json hyperschema?

A hyper-schema implementation takes an LDO, an instance, and (for forms) client data, and produces a resolved link and (for forms) payload.

The caller of the hyper-schema implementation needs to know (with assistance from the URI scheme, targetHints, and headerSchema) what operation(s) it can perform.

• For an HTTP GET, this just means resolving the URI (possibly using client input with hrefSchema)
• For an HTTP PUT, this additionally means using the (presumably modified) instance data as the payload. targetSchema and targetMediaType can be used to indicate what can be changed in the instance, and within what constraints
• For an HTTP PATCH, the PUT approach is modified by a PATCH media type, which could be advertised with accept-patch in targetHints
• For an HTTP POST, rather than using the instance, client data is needed for the payload, as described and validated by submissionSchema and submissionMediaType (or, I suppose, by accept-post in targetHints since that header is now registered)
• For an HTTP DELETE, as with GET, hrefSchema is the only input that could be relevant.

A schema could use allow in targetHints to explicitly advertise method support. Or one can usually assume that if there is a submissionSchema then in an HTTP environment, the target resource supports POST. Or at least sometimes does- runtime responses always take precedence over hyper-schema.

---

Hyper-Schema has come a long way in the last year since I met you and many other WoT folks in Santa Clara. I think the bulk of the work on draft-06 actually happened right after that, and in my opinion, draft-07 is the first reasonably feature-complete, usable draft of Hyper-Schema with specification that is testable for compliance. I am, of course, biased.

There are some examples in the draft-07 spec- much more comprehensive in nature than in past specs. As for more real-world examples, I'll let you know when I have it running :-)

[egekorkan]

It seems that everything except the JSON Hyper Schema discussion has settled down. I am not aware of anyone working with JSON Hyper Schema in the group and it would be interesting to evaluate it. I would propose to close this issue if no objections by 25.01.2020 and open a new one titled "Analyzing JSON Hyper Schema" or something similar.

[handrews]

@egekorkan as an update on Hyper-Schema:

We spent the last two years focusing on getting JSON Schema Core and Validation to something resembling a feature-complete status. Most importantly, this meant a formal, interoperable extension mechanism for adding further vocabularies (including by groups other than the JSON Schema project). In this approach, Hyper-Schema is one such vocabulary (that does happen to be owned by the JSON Schema project).

In addition to allowing further vocabularies, you can independently restrict the syntax of the vocabularies in use. So if, for example, you do not want to allow the array form of the type keyword, you can restrict that using a meta-schema while still indicating that you are using the standard keyword semantics for type with a single value.

We accomplished all of this with JSON Schema draft 2019-09. (We changed to date-based identifiers with this draft). The vocabulary concept is sufficiently complex that we expect to finalize it after feedback from the field, but we think that fundamentally it will work (there is one implementation that was done while we wrote the spec).

A major goal was to align with OpenAPI, which was using a modified subset of a very old draft, with some extensions that did not fit well with JSON Schema's approach. Like the TD, the OAS document embeds JSON Schemas inside of a JSON or YAML document that is not itself a JSON Schema. It also has code generation and documentation generation requirements that are not well-served by the validation keywords alone.

I am pleased to announce that the OpenAPI Specification v3.1 is adopting JSON Schema draft 2019-09 (or possibly a slight clarification/bugfix update that we will produce in the next two to three months).

Hyper-Schema received only minimal attention in 2019-09 (rel can now take multiple values as an array, with semantics matching space-separated values in the HTTP Link header). I think it will be interesting to come back to Hyper-Schema once we have folks producing updated 2019-09 (or 2020-03 if we do that update) tooling intended for use with both standalone JSON Schema and OpenAPI. OpenAPI is one of the largest users of JSON Schema, and the fact that they were on an old draft was one of the major factors holding adoption of newer drafts back.

Since OpenAPI is a static design-time system that intentionally does not address runtime hypermedia, I am hoping to revisit Hyper-Schema as a runtime solution now that it could be used in conjunction with OpenAPI, rather than being positioned as competition.

We also expect extension vocabularies to be developed to support code generation and other use cases, which again should be usable both inside and outside of OpenAPI.

I think that this general approach might be suitable for the TD, as it specifically takes the idea of integrating JSON Schema with other projects and use cases than straightforward standalone validation.

[egekorkan]

I will be closing this issue and reopening a new one with regards to JSON Hyper Schema by following the comment just above.