[Discuss] Execute HTTP requests internally

[streamich]

For Dashboard and Canvas we need to batch multiple requests and stream responses back. To do that one approach would be to implement batching on HTTP level, where any HTTP request can be put into a batch, that batch is sent to the sever and server executes internally each individual request. If we go with this approach, we would need from platform:

• Allow plugins to execute locally HTTP endpoints without going through HTTP.

Optionally:

• Handle authentication headers automatically.
• Handle spaces automatically.

Related: #44393

[elasticmachine]

Pinging @elastic/kibana-app-arch (Team:AppArch)

[elasticmachine]

Pinging @elastic/kibana-platform (Team:Platform)

[mshustov]

For what kind of endpoints you want to use it? Do we know all of them? I suppose so. Why plugins cannot declare their dependencies explicitly and use a plugin API to perform batching? Allowing a plugin to call any endpoint creates hidden dependencies between plugins. That's exactly one of the reasons why the new platform mechanism was introduced.

[streamich]

Why plugins cannot declare their dependencies explicitly and use a plugin API to perform batching?

They can, that is what we are currently doing. My understanding is that the idea behind batching on HTTP level is that it is more powerful. /cc @ppisljar @chrisdavies

Allowing a plugin to call any endpoint creates hidden dependencies between plugins. That's exactly one of the reasons why the new platform mechanism was introduced.

Plugins already can hit any endpoint.

[pgayvallet]

'Batching' http requests has subtle implications that makes it difficult to implement at a whole platform level.

It's one thing (relatively easy) to implement manually on a specific endpoint, by receiving a list of 'request' or 'action' in the payload and process them, then return a 'batch' response handled by the client side, as it's manually handled by the endpoint and it actually just a classic http call to an endpoint.

Trying to implements this mechanism at at lower level introduces a lot of questions on the actual router implementation, as it's very rare for http server/router mechanism to totally decorrelate the transport layer from the protocol layer of what we call http (and it's definitely not the case of HAPI)

• regarding middlewares / lifecycle interceptors: If the 'batch' requests target different endpoints, How do you decide which middleware should be applied? I guess we want each 'batch' request to get intercepted the same way they would if they were performed independently. Unfortunately this is not possible without separating the transport layer from the protocol layer, which is an extremely heavy change (basically that means stopping using HAPI for started)

• how do you cancel a specific request of the batch from the client side?

• This is a complete different API on the client-side too

To summarize, I think that can't really be achieved on the router/core level, and that if some teams/features requires this, they should implement it manually at the endpoint level.

[mshustov]

They can, that is what we are currently doing. My understanding is that the idea behind batching on HTTP level is that it is more powerful. /cc @ppisljar @chrisdavies

What advantages does it provide?

Plugins already can hit any endpoint.

It doesn't mean they should. I assume we will introduce public/private API declaration for endpoints soon or later. Although it requires a separate discussion about the public / private API separation, contract testing, and auto doc generation. Using plugin API via contracts is the best option at the moment.

[streamich]

how do you cancel a specific request of the batch from the client side?

I don't think you can in current Kibana, as it can be run in cluster mode and then you don't have an ability to hit specific server.

What advantages does it provide?

That you can call any URL.

[ppisljar]

when kibana is in cluster mode, is there any way to communicate between nodes ?

[mshustov]

Un-assigning myself as I'm not actively working on it.

[ppisljar]

closing this due to inactivity, please re-open if its still relevant