Overview - all TD implementations

[mlagally]

The attached CSV file all-TD-assertions.csv is a combination of the implementation reports that are contained in the testing/inputs/results directory.

The list of all assertions is the combiation of all assertion ids from the individual contributions.
Cells marked as XXXX are assertions that are not contained in the respective input.

A couple of observations:

• The TD specification contains 165 assertions (all-assertions.csv), however many input files contain >400. Why?
• Some responses look as if they were submitted during the 1.0 version of the specification and never were updated. It is questionable whether these should be used as implementation proof for the TD 1.1 specification.
• Several distinct responses were provided by the same companies and should not be counted as individual contributions. There should be rather a single combined implementation report from each company.
• Some responses by companies that are no longer members of the WG are also present - these should be excluded.

[sebastiankb]

Many thanks for this overview. Interesting to see which feature was implemented by which tool/lib. However, the tables seems to incomplete since many assertions are missing. May I ask how you obtained this result?

Regarding your observations:

The TD specification contains 165 assertions (all-assertions.csv), however many input files contain >400. Why?

As mentioned above, in your table there are around 300 assertions missing. A complete overview of all TD assertions can be found here.

Some responses look as if they were submitted during the 1.0 version of the specification and never were updated. It is questionable whether these should be used as implementation proof for the TD 1.1 specification.

TD1.1 claims to be backwards compatible. So the best test is to use TD1.0 instances if they are compliant with TD1.1. Also this was agreed in the previous calls.

Several distinct responses were provided by the same companies and should not be counted as individual contributions. There should be rather a single combined implementation report from each company.

There was a decision a while ago to test and to list the individual implementations. Please note that the TestFest in March run this mode already.

Some responses by companies that are no longer members of the WG are also present - these should be excluded.

In general, it is not necessary to be a member of the WG to participate the TestFest. I think, it is fair to mention the implementations who contributed to TD1.0 and were used to test backwards compatibility.

[mlagally]

@sebastiankb

Many thanks for this overview. Interesting to see which feature was implemented by which tool/lib. However, the tables seems to incomplete since many assertions are missing. May I ask how you obtained this result?

Regarding your observations:

The TD specification contains 165 assertions (all-assertions.csv), however many input files contain >400. Why?

As mentioned above, in your table there are around 300 assertions missing. A complete overview of all TD assertions can be found here.

I generated the list of assertions with the script from @k-toumura , which searches through the spec for the RFC-2119 markup.

[sebastiankb]

I generated the list of assertions with the script from @k-toumura , which searches through the spec for the RFC-2119 markup.

Many thanks for clarification. I'm not 100% sure how the script works from Toumura-san. The script that was used for finding all TD assertions can be found in the TD testing folder.

I will remove the "by CR transition" label since the overview is not part of the TD spec or IR.

[k-toumura]

The script is for Architecture document, and can only extract DIVs of class rfc2119-assetion. So, it cannot extract rfc2119-default-assertion(21 assertions) and rfc2119-table-assertion(113 assertions) which are included in the Thing Description spec.

[mmccool]

What @k-toumura said. The script is overlooking assertions using alternatives to the rfc2119-assertion class for table entries. As for XXXX assertions: lots of reasons for these, including manual assertions submitted for TD1.0 that are no longer applicable (they MIGHT be, but there were a lot of id changes, unfortunately, which broke our ability to associate the old and new assertions without some extra work). Note that automatic test results will be up-to-date.

In general we agreed to include the older inputs as test cases. Testimonials will be a different matter. For the final IR (this is a draft) we probably want to add some text, and maybe some tables or extra table columns, to explain the use of older inputs. Making such improvements to the IR though can wait until after the CR transition; the IR is marked as a draft and is not a blocker for CR transition.

[sebastiankb]

from today's TD call: @mlagally what is the status about this PR?

[mlagally]

The all-assertions file was incomplete, so I removed it.
the remaining cross reference adds value and should be merged.

[egekorkan]

We will look at it in detail next week