[Propose] Kuasar Shim and Domain Sandoxer Server

[Iceber]

What would you like to be added?

Currently, the various sandboxers in kuasar are running as domain servers resident in the nodes.
The communication between containerd and sandboxer is done by modifying the logic of the Sandbox and Runtime for Containerd, and the impact of these modifications on Containerd is huge.

Can we achieve the goal of kuasar without modifying Containerd, or without making huge changes?

The goal of kuasar as I understand it should be:

1. Ability to support multiple sandbox types
2. No large number of long-running shim processes

Currently kuasar removes long running shim by sandboxer of domain server, kuasar-io/containerd links sandboxer server by specifying sandboxer address in configuration and modifies Sandbox interface and runtime logic.

[proxy_plugins.vmm]
  type = "sandbox"
    address = "/run/vmm-sandboxer.sock"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.vmm]
  runtime_type = "io.containerd.kuasar.v1"
  sandboxer = "vmm"
  io_type = "hvsock"
[proxy_plugins.wasm]
  type = "sandbox"
  address = "/run/wasm-sandboxer.sock"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.wasm]
  runtime_type = "io.containerd.wasm.v1"
  sandboxer = "wasm"

We can actually keep running sandboxer as a domain server, but we don't need to make any changes to Containerd (of course there may be minor changes, but this is much smaller than kuasar-io/containerd).
I think Sandbox and the core is the Sandbox server, which is a very flexible and versatile design, and I think we should try to use it as much as possible.

We keep the lightweight shim binary for containerd to call, and the shim start command will connect to the domain server to request the address of the task server and sandbox server, and return that address to containerd's runtime(shim manager).
Unlike containerd-shim-runc-v2, the start subcommand of kuaras shim(eg. containerd-shim-kuasarvmm-<version> or containerd-shim-kuasarwasm-<version>) does not fork itself to start the task server and sandbox server, they will exit after returning the address and it does not run for long.

This is already used in runwasi

The Shim is no longer a shim for containerd and runc or other underlying runtimes, but a shim for containerd and sandboxer server

Interacting with the domain sandboxer server through shims that do not run for long periods of time is fully compatible with the current sandbox and runtime design, and ensures that our goal of not having a lot of long-running shim processes is met.

---

BTW, Also I found a new question, what if we update the sandboxer server with minimal impact, since sandboxer manages all the underlying container processes, it might be important how we can upgrade it more smoothly.
If we have the perfect upgrade solution, maybe we can have all sandboxers merged into one kuasar-sandboxer domain server

Maybe we could use wasm to implement the core logic in kuasar-sandboxer and plug in the specific runtime implementation through wasm, that's just an idea.:rofl:

Why is this needed?

We can actually keep running sandboxer as a domain server, but we don't need to make any changes to Containerd (of course there may be minor changes, but this is much smaller than kuasar-io/containerd).
I think Sandbox and the core is the Sandbox server, which is a very flexible and versatile design, and I think we should try to use it as much as possible.

[kuasar-io-dev]

cc @abel-von @Burning1020

[Burning1020]

Brilliant idea! Thanks!

Actually we have a shim crate for keep compatibility with containerd. However, our current shim is just a proxy service redirecting rpc request from containerd to sandboxer, so it needs long running on the host, which is not a permanent solution.

I think your propose is a possiable way to evolve kuasar shim. The removel of shim maybe the final goal of kuasar, we could make some steps towards it, like what you have proposed. I'm going to arrange a meeting discussing about this.

As for upgrade scenario, sandboxer will persistently store sandbox info to the disk, and reload it after restart. So there is no worry for the lost. But the service can't available during upgrading as contaienrd can't either.

[Burning1020]

We'd like to take this propose as the next evolution direction of kuasar shim, and we also need to talk about our project on containerd community meeting.

[Iceber]

Thanks, I would also love to have more detailed details on how to move forward with containerd/containerd#7739 (e.g. completing each task of the issue design).
I'm probably most interested in how to better split sandbox service address and task service address, and I'd like to know what others in the community suggest