Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ADR #8: Container image expansion #3

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

ADR #8: Container image expansion #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions 0008-container-image-expansion.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Buildbarn Architecture Decision Record #8: Container image expansion

(c) Copyright 2022, Zenseact AB. All rights reserved.

Author: Fredrik Medley<br/>
Date: 2022-04-21

# Context

When running remote execution, the clients are usually not providing all the files as inputs. Instead, it is assumed that the action is run in a specific environment. The defacto standard is to specify a container image as part of [the `platform` field](https://github.com/bazelbuild/remote-apis/blob/2af1c432ce11b01e46cf70d7933f71a3310282d6/build/bazel/remote/execution/v2/remote_execution.proto#L520) of the action, using the key `container-image`.

The problem comes to the split of responsibility where the client specifies the wanted container image and the server needs to prepare runners with that container image. In the general case, the server side cannot predict all different container images the clients would like to use.

# Proposed change

In this ADR we propose to add a new `bb-container-expander` service, in front of the `bb-scheduler`. For each specified `container-image` (yes, the protocol accepts multiple entries) among the platform properties, the [input Merkle tree](https://github.com/bazelbuild/remote-apis/blob/2af1c432ce11b01e46cf70d7933f71a3310282d6/build/bazel/remote/execution/v2/remote_execution.proto#L468) will be extended. The resulting action will not specify any `container-image` at all. The worker will also have to employ [`chroot`](https://github.com/buildbarn/bb-remote-execution/blob/c4bbd24a8d272267f40b234533f62c8ccf24d568/pkg/proto/configuration/bb_runner/bb_runner.proto#L36-L39) when executing the actions, optionally moving the original input tree into a specific directory.

<p align="center">
<img src="images/0008/bb-overview.png" alt="Buildbarn with the proposed bb-container-expander"/>
</p>

The input tree might become very big. This might lead to long times to hard link files to the runner from the worker cache. This can be mitigated by using the FUSE option for the runner build directory.

The process of expanding a container image will be to:
1. Download the metadata for the image.
1. For each layer, download the layer and upload it as a Merkle tree to the content addressable storage (CAS).
1. Merge Merkle trees for each layer in order.

Each step will be cached in the action cache (AC), using a "fake" action. It can look like setting the command line as `- bb-container-expander download-metadata ...` which should not collide with any actions provided by the client.

Assuming that the container image files and the client provided inputs are generally located in different directories, most of the Merkle tree merging will happen close to the root level and therefore not consume much resources. Merging the layers will go deeper in the structures, so a local AC and Merkle tree cache (CAS) is probably necessary.

# Other considered solutions
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think there are a couple of other options we should consider/document:

  • Making Bazel itself responsible for emitting actions that have the right shape.
  • There is the ActionRouter API in bb-scheduler, which can already be used to do rudimentary matching/rewriting of platform properties. Maybe that subsystem could be extend to also be able to interpret container-image and rewrite the action before it's submitted to the worker?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding Bazel, that would probably be the "correct" place to put it. Then Bazel itself could use the information when running locally. Maybe specifying a root_directory label in toolchains and/or platforms would be the solution. The problem is that the way Bazel put outputs would make it very impractical to specify a root tree. In practice, the only kind of root file system I can see is a container image. Referencing a container image built with Bazel would be something but I haven't looked into rules_docker yet.

Anyway, other remote executors supports the container-image platform property out of the box, so the feature might still be valuable within Buildbarn.

Am I missing something? I feel that a Buildbarn implementation would be easier than adding it to Bazel but I feel that I cannot find a good enough argument for not putting it in Bazel (more than too much effort).

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with ActionRouter is that it only extracts the keys, my plan was to rewrite the action, or at least the input root of it. Now, the action is already rewritten before sent over to the executor, actionWithCustomTimeout. Shall the ActionRouter be taught to return the actual_input_root_digest as well?

Copy link
Member

@EdSchouten EdSchouten May 6, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should just reach out to the Bazel folks and ask them what their vision is on this. If they are interested in solving this inside Bazel, then we should help them out. If they despise the idea, we could support it on our end.

With regards to ActionRouter: indeed! We should let that return a new input root digest.


If the set of container images to be used is given, autoscaling in Kubernetes can be used to scale certain images up and others down, depending on the load. This helps to keep the locality of the files, where all actions using the same image will be scheduled on the same runners.

To support dynamic images, another service would have to listen to the scheduler for operations that do not have any matching runner and quickly spawn such runners.

This solution was not chosen as it gets limited to Kubernetes, it doesn't work in docker-compose or on a bare deployment. It might also need to be tailored for different Kubernetes suppliers, adding maintenance work.
Binary file added images/0008/bb-overview.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.