Look into updating the audit files

[starksm64]

The TCK has various tck-audit-*.xml files that conform to the http://jboss.com/products/weld/tck/audit schema that providing a mapping from specification assertions and associated text to the actual unit test. This has gotten out of date, and it should be a goal to bring these files back into alignment with the specifications.

With the move to asciidoc, it may be possible to create a converter that allows for the generation of the audit files.

[manovotn]

Thanks for filing the issue. Had it on my TODO but went offline earlier yesterday.

To fill in some details:

• TCKs audit files were for CDI spec, interceptors and managed bean definition although we are mainly looking at having it working for CDI
  • They can be seen here
• Upon building TCKs, a file is generated locally with a name of Sections.java that links the audit XML file with annotation values used on actual tests
• For TCK 1.2 and 2.0 we specifically generated an HTML version of spec with links to TCK. An example of such doc can be seen here
  • So you could either browse this doc or you could just open up the audit file and search there and then grep TCKs for needed assertions

The separate project that is used to generate the links between tests and audit file is this one - https://github.com/jboss/jboss-test-audit. To my knowledge, it was only ever used by CDI and bean validation specs but it shouldn't be an issue to export it under jakarta repos if we need/want to.

Personally, big +1 for putting this back on track. This makes working with TCKs and looking for certain tests much easier. Although it will be quite some effort due to ongoing re-shuffling of spec due to Lite addition.

[starksm64]

So I have done some basic testing of how one might create the audit-*.xml files from the asciidoc. It seems doable, but one would have to adopt a convention for identifying the test assertions using asciidoc roles/attributes and should probably also include an anchor to be able create a comprehensive cross references of assertions.

The big work will be updating the tests to use the updated @SpecVersion and @SpecAssertion or matching the generated files to their usage. I don't quite have my head around the entire process yet.

[Ladicek]

I have also thought about how spec<->TCK cross-referencing could work. Apparently, Asciidoctor can be extended using macros, so we could define syntax like specref:foobar[This is a sentence in the specification.]. That's not the biggest problem though.

The biggest problem is that it's a kinda chicken-and-egg situation. If you have the spec text annotated with macros like specref, you can then process the Asciidoc and generate a set of Java classes that can be used in the TCK (so that the references in the TCK aren't strings, but Java enums). But then, to build the TCK, you need a build of the spec, and to build the spec with links to TCK, you need a build of the TCK. If "building the spec with links to TCK" has always been a 2nd phase, after the normal spec build and then TCK build, then that's fine, but if not, that would be a significant regression.

Oh and personally, I don't really like how the spec with links to TCK looks right now. IMHO, it should look like the normal spec looks, except each sentence (or block) that has a corresponding TCK test, would be enclosed in some parens and after those parens, there would be a small link to the test (not class, but directly the method, that should be easily possible).

[manovotn]

Oh and personally, I don't really like how the spec with links to TCK looks right now. IMHO, it should look like the normal spec looks, except each sentence (or block) that has a corresponding TCK test, would be enclosed in some parens and after those parens, there would be a small link to the test (not class, but directly the method, that should be easily possible).

The original idea was to deliberately ship two variants of the spec doc as most people don't need/want to see links to TCKs while browsing the spec. And I think that was a good idea. It also allows you to perform this extra spec doc build after you have both, CDI and TCK built (if I understand your chickens and eggs correctly :))

Also note that:

• The audit XML file is not exactly 1:1 with spec text; for example some sentences were split into two or more separately testable assertions
• Some spec assertions are not testable at all (those are now marked with testable="false" in the audit file

[Ladicek]

Yea I didn't think the only spec variant should be "with TCK annotations". There should of course always be a plain spec. I thought about it more and my chicken-and-egg problem can be easily resolved by putting the TCK in charge of rendering the annotated spec version.

The workflow would look something like this:

1. HUMAN: The spec source is annotated using a specref macro (e.g. specref:foobar[This is a sentence in the specification.]).
2. MACHINE: The spec build process would, in addition to rendering the spec, also collect all the specref macros and generate Java source code that the TCK source can use to reference the specification. The specref IDs have to be unique in an AsciiDoc section, not globally. Also the spec build process has to compile that generated Java source code, and generate a -sources.jar with the AsciiDoc sources of the spec.
3. HUMAN: The TCK uses the @SpecReference annotations to refer to the spec, using enums generated in the previous step.
4. MACHINE: The TCK compilation is augmented using an annotation processor that collects all the @SpecReference annotations and generates a JSON file that lists all the @SpecReference occurences in Java source code (and maps them to AsciiDoc specrefs, that must be possible somehow). Actually multiple JSON files, one for each TCK module.
5. MACHINE: After all TCK modules are built, an extra module is built that renders an annotated specification, from the -sources.jar of the spec and the JSON files generated in previous step.

I might be missing something obvious, but this seems like a much more straightforward solution than what we have right now?

[Ladicek]

I was able to build a proof of concept. Here's how the outcome looks like: https://ladicek.github.io/cdi-tck/#legal_bean_types

The proof of concept code itself is here: https://github.com/Ladicek/specref

The required changes to CDI spec are here: https://github.com/Ladicek/cdi-spec/commits/specref

And the required changes to CDI TCK are here: https://github.com/Ladicek/cdi-tck/commits/specref

[starksm64]

@Ladicek So you also created a io.smallrye.specref-core repo as well, because I'm not finding that in the io.smallrye org. Is that an asciidoc extension or converter?

[Ladicek]

@starksm64 I used the io.smallrye package name, because I thought that's where we would put it, but I didn't make this official in any way yet. So no, there's nothing in the smallrye org. It's a proof of concept.

The specref:abc[Some text in the spec] is an Asciidoctor inline macro, and it's handled by an Asciidoctor extension. Actually 2 extensions -- one just passes through the Some text in the spec and collects the abc identifiers (that's used when building the spec), and the other renders the Some text in the spec annotated with a list of corresponding TCK tests (that's used when building the annotated spec as part of the TCK build).

(In between the 2 Asciidoctor extension invocations, there's a Java annotation processor. It actually works as described in #259 (comment) -- I just proved that the concept described in that comment can actually work.)