[HTTP] Support public HTTP API availability statuses

[jloleysens]

When a Kibana API is made public there is no way to capture it's stability: e.g. "experimental" or "stable". Being able differentiate public APIs in this way provides some freedom to run experiments and learn about real usage before committing to supporting an API for +18 months.

Further, having this kind of metadata present in code would enable us to communicate the overall API level of stability in docs, requests and responses.

The following sections are some thoughts and discussion we've had on this topic. Additional input is welcome!

Elasticsearch-specification approach

We can piggy back off the Elasticsearch-specification. It documents API "stability" as one of experimental, beta, stable. For example. This value could be surfaced in request responses as a header, something like x-availability-stability: experimental to further increase visibility of stability.

The break-when-graduate approach

An extension to ES-specification approach: instead of only sending back a header we can encoding stability into something like the version or URL path. Anything that forces a break once the API moves from experimental to stable. This may make it easier to choose to end the experiment by removing an API entirely but could also reduce involvement if users know APIs could vanish whenever we choose.

Overall the intention is provide a way to avoid future experimental APIs from becoming relied upon by end users in unintended ways.

There are a couple of mechanisms for forcing breakages that I could think of:

1. Encode availability in the version, something like 2023-10-31.experimental
2. Encode availability in the path, something like /api/experimental/<rest-of-path>

Generally, avoid experimental APIs

Not all APIs are equal and it could be dangerous to release certain capabilities even under the "experimental" label (thinking specifically of SO APIs). This is where the "break-when-graduate" approach could be helpful because users more explicitly opt in to an experiment.

Ideally there are ways of gathering data other than releasing new APIs we are not sure we'll support.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[jloleysens]

Related #105692

[TinaHeiligers]

@jloleysens we need to pick up the discussion for upcoming breaking changes

[TinaHeiligers]

related #192292

[rayafratkina]

Will tackle this work as part of enabling new API versions