Integrate bazel remote cache server with the bazel build POC

[mistic]

After setting up the kbn packages to build with bazel (#74104) we should experiment to integrate it with a remote cache server.

[elasticmachine]

Pinging @elastic/kibana-operations (Team:Operations)

[joshdover]

I've been thinking about this some more and while I'm really excited about Bazel in the long-term, I'm wondering if we should do something really simple in the short term for the @kbn/optimizer in development.

We already have cache keys produced by @kbn/optimizer so I wonder if we should go ahead and setup a really basic blob store that the optimizer can lookup pre-built assets from. I think the biggest hurdle would be figuring out how to do this in a safe way so that only trusted machines are allowed to write to the cache (maybe only CI?), but otherwise it may be simple enough to setup a storage bucket in GCP.

It could work something like:

• Take our existing cache key (which is a JSON doc), and produce a hash with sha256
• See if an object with that key exists in the cache:

  GET https://storage.googleapis.com/storage/v1/b/kibana-dev-optimizer-cache/o/<sha256hash>.zip
  

• If it exists, download and extract all the bundles for that plugin into it's target/public directory
• If not, build the plugin

In CI, we would have the appropriate credentials for putting new items into the cache.

cc @spalger

[spalger]

It's definitely something I've suggested to folks before, I don't think it would be that hard to build, but I haven't seen any interest before now in a feature that will likely rely on some infrastructure to get right.

Converting the cache key to make sure that it is totally reproducible and not reliant on the OS or repo location would be the first step.

[joshdover]

I think we're getting to the point where we've squeezed what we can out of the optimizer and other than removing legacy plugin discovery, I'm not sure there's a lot of low hanging fruit left to improve the start up time of Kibana when switching branches.

Could be a really good Spacetime project that would have a big impact.

[tylersmalley]

I agree that this could be a big win if we get the cache key right.

@spalger, would you mind creating an issue to track adding remote caching to kbn/optimizer and include any changes you see us needed in terms of generating a cache key? I would rather we move the discussion there than take over this one related to Bazel and the packages POC.

[mistic]

I think at this point in the context of the POC we already did a lot of tests with the remote cache server. More info could be found on #69706 (comment) in the Remote Cache section