Inhibitor support for graceful node shutdown

[tanvp112]

Hi,

I wanted to do some tests for graceful shutdown. Noticed that systemd-inhibitor was not included in the kind image. The command systemd-inhibit --list would return error due to no systemd --user is running.

Is this currently not supported or am I missing any configuration here?

Thanks.

[stmcginnis]

I'm not sure how that would work with kind.

Graceful shutdown happens when the host is shutting down. IIRC, kubelet gets registered with systemd-inhibitor so that when shutdown is happening, systemd will wait a given period of time for any registered processes to exit before continuing with the shutdown.

Since kind nodes are containers, they do not have their own systemd process, and shutting down is done by stopping the container. So it wouldn't ever hit this condition.

Can you explain a little more about what you are trying to test here?

[tanvp112]

They do have systemd process but was reasonably slim down, I reckon systemd-inhibitor-locks can be added similarly? https://github.com/kubernetes-sigs/kind/blob/main/images/base/Dockerfile#L49

I am guessing system shutdown event can be handled? https://github.com/kubernetes-sigs/kind/blob/main/images/base/Dockerfile#L241

Example I have a long running pod on a spot node that about to be reclaim. I want to test to ensure the pod can stop within the time limit, including the volume attached (not the default local storage).

[stmcginnis]

Ah, sorry, I should have done a little more research before commenting. You are totally right.

It may be possible to add systemd-logind to the base image. That should pull in systemd-inhibit. I'm not sure when I can get to it, but I will try later. You could also try building your own base image with that modification, then create you own node image.

[aojea]

I don't know if kind is the right project to test node shutdown, @BenTheElder you are much knowledgeable on this area PTAL

[tanvp112]

Graceful shutdown or non-graceful shutdown of node can be simulate with docker container stop kind-<name> --time <duration>, but it needs inhibitor lock to take advantage of the time allowance. Given default KIND image already has systemd running, what's the reason KIND is not the right project to test node shutdown?

[aojea]

kind does a lot of hacks to emulate a VM, are you sure all the behaviors triggered by a VM shutdown can be simulated with docker container stop? ... kind does not have dbus per example

[tanvp112]

Yes. But I think the bigger issue here is DBus is needed for kubelet to obtain the lock inside the node.

[aojea]

so, it is worth it to add it to kind or just test this in a vm? the later seems more appropriate to me

[tanvp112]

Why not? Note it is not impossible to add dbus to container. rke2 got this done for sometime already. Worth or not, guess it depends on how difficult to maintain for long run, certainly not because of dbus or systemd.

Feel free to close this request if graceful shutdown is too much a trouble that KIND doesn't see a value, you are certainly right that there are many other options out there.

[aojea]

We don't have lifecycle on the container, and the VM emulation fails in areas as the abstraction of the host resources... Adding something incomplete that will not provide full coverage seems too much cost for small roi ...

[BenTheElder]

As a general statement: System integration / node level isolation is going to be problematic with kind, because fundamentally we're sharing a host kernel. It's pretty OK for testing distributed behavior, manifests, controller logic, etc, but when we get into kubelet/node stuff it may or may not be appropriate. We try to enable these but it's sometimes pretty messy and it hasn't been the main focus. (see e.g. #1963 and the problems there)

Why not? Note it is not impossible to add dbus to container. rke2 got this done for sometime already. Worth or not, guess it depends on how difficult to maintain for long run, certainly not because of dbus or systemd.

I'm not that familiar with dbus, but this looks plausible to me. We'd have to investigate this more, it needs to be isolated from the other containers / host.

We don't have lifecycle on the container,

I think this is the bigger problem, we usually have users simulate node addition/removal using taints which is sufficient for testing most applications. For testing Kubernetes with disruptive node behaviors we typically still use cloud VMs where we have better isolation and control.

kind only has "create cluster" and "delete cluster" currently (and ... introducing other mutations would get pretty complex)

Inhibiting shutdown here would be somewhat at odds with any container deletion behavior (e.g. this thread #2272 (comment)), but that's not necessarily a deal-breaker for tests if we inhibit for less time than the timeout, or simulate shutdown signal without actually calling kind delete cluster (maybe we signal systemd instead?)

More importantly: If you're writing tests for graceful shutdown in github.com/kubernetes/kubernetes, I'd say these should probably be under node_e2e and SIG node pretty much only supports node_e2e with kube-up.sh currently, to my knowledge.

I'd love to enable these someday, but I'm not sure if that's even something the SIG node maintainers are interested in, overall, and if not then we should probably continue to focus on cluster e2e (test/e2e).

If I've misread and you have another use case please elaborate 😅 .

[BenTheElder]

(also this week is KEP freeze amongst other things so please bear with response times ...)