[http] Relax API access restrictions

[TinaHeiligers]

Relates to #151940.
Change the default API access flag to internal to give API owners more time to properly handle their API requirements.

Having APIs default to internal is a temporary measure to relax the restrictions implemented in #151940.

Change the way the RouteConfigOptions access parameter is set when not specifically configured to default to internal.

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[lukeelmers]

Just adding a few notes about the intended behavior based on my understanding of our discussion:

• the access option is only enforced when a config flag is enabled in serverless
• so we will default access to internal, meaning that in serverless all APIs will be treated as internal by default
• and in self-managed, nothing will change: these apis will still technically be classified as access: 'internal' based on the route config options, but since nothing is actually enforced, it won't affect end users. we will need to be careful not to introduce any other functionality in self-managed that relies on this access parameter to avoid confusion.
• any APIs which have already been set to explicitly be access: 'public' will need to be removed or changed to internal or they will continue to be public in serverless
• Elastic owned APIs calling into internal kibana routes externally will need to use the x-elastic-internal-origin headers (as is already the case today), but:
• devs will still have the ability to explicitly set their APIs to be public in serverless if they are willing to own the maintenance of these
• long term: whenever we plan a 9.0 self-managed release, we should discuss making this the behavior global across self-managed and serverless. ideally every http api registered in kibana is internal unless someone opts-in to make it public.

[TinaHeiligers]

any APIs which have already been set to explicitly be access: 'public' will need to be removed or changed to internal or they will continue to be public in serverless

Are we sure about that? Teams who have already explicitly set access did so for a reason. IMHO, we should leave those as they are and rather ask teams to review their decision, specifically w.r.t changing 'internal' APIs to 'public', not the other way round.

[lukeelmers]

we should leave those as they are and rather ask teams to review their decision

++ This is what I'd propose too. I was simply pointing out that if the guidance eventually becomes "don't expose any public HTTP APIs unless absolutely necessary", anybody who has explicitly marked an API as access: 'public' will need to revisit that decision. But this should be up to individual teams... I only mention it here to clarify that APIs don't all suddenly become internal when we finish this work; there will be some lingering public APIs which teams must review.