Develop strategy for differentiating stable vs. experimental conventions, publishing stable artifact

[jack-berg]

The semantic conventions contain a mix of stable and experimental conventions. This maturity level is denoted in the generated markdown (e.g. see stable resource conventions), but not in the yaml representation used to generated java classes (e.g. see resource conventions).

Semantic conventions will always be a work in progress to some extent, with some domains / attributes marked as experimental. We need a strategy for being able to release the stable conventions in a stable way with backwards compatibility guarantees, while also allowing for experimental conventions to evolve and break over time.

[trask]

once HTTP semconv is stable, and we mark HTTP instrumentation as stable, it would be nice if that stable instrumentation doesn't have an unstable dependency

we don't technically require this, since we've had the pattern of unstable implementation dependencies before, but this one feels a bit more problematic because I think it's going to be very common for end users to pull in the semconv artifact

I think this would require doing one of these:

• mark the whole semconv artifact as stable
• split the semconv artifact into unstable and stable artifacts
• inline the stable semconv into the instrumentation-api

[jack-berg]

What do you think about a single stable semantic conventions artifact, but with an io.opentelemetry.semconv.experiemental.* package to house experimental attributes? Sort of like the *.internal.* packages in core and instrumentation, but "experimental" better articulates that these are meant to be used but subject to change.

[jack-berg]

Proposal based on conversation from 12/07/23 SIG:

• Publish two artifacts, io.opentelemetry.semconv:opentelemetry-semconv and io.opentelemetry.semconv:opentelemetry-semconv-incubator (to mirror naming of opentelemetry-java-instrumentation/instrumentation-api-incubator.
• io.opentelemetry.semconv:opentelemetry-semconv contains only stable conventions, and would have strong backwards backwards compatibility guarantees using japicmp once we're comfortable with the generate logic
• io.opentelemetry.semconv:opentelemetry-semconv-incubator contains experimental conventions and we have no backwards compatibility guarantees and no use of @Deprecated annotations. As semantic-conventions change, we reflect those changes as is. Since we won't have any deprecation policy, breaking changes will be common in practice.
• The strong backwards compatibility guarantees for io.opentelemetry.semconv:opentelemetry-semconv allow instrumentation to safely add dependencies, since transitive version conflicts can be resolved by using the latest version without error.
• Since there is no compatibility guarantees or attempt to solve the diamond dependency problem in io.opentelemetry.semconv:opentelemetry-semconv-incubator, native instrumentation should not take a dependency on this, since transitive version conflicts will be prone to producing unresolvable compilation errors. We document this behavior and instruct any native instrumentation which wants to encode experimental semantic conventions to copy the conventions into their source.

cc @lmolkova

[jack-berg]

Additionally:

• Include the schema url version in the name of the constant for schema url. We currently have a fixed SCHEMA_URL whose value changes version to version. For io.opentelemetry.semconv:opentelemetry-semconv, keep a running tab of all previously published constant names (i.e. SchemaUrls.V1_23_1, SchemaUrls.V1_24_0, etc). This will ensure that when transitive version conflicts resolve to the latest, code depending on an earlier schema url will continue to compile. For io.opentelemetry.semconv:opentelemetry-semconv-incubator, do not publish any schema url. The incubating module should depend on the stable artifact, which will have a running tab of all schema urls. No need to repeat work.