Using Wants not Requires on configured.target

[tjkirch]

(See #96)

The kubelet, docker, and containerd service files have dependencies on configured.target that are expressed with Wants instead of Requires.

This means that if configured.target fails at boot, kubelet and friends will start, even though we know we haven't generated the configuration files they need.

(This was done to avoid an even worse problem - if we used Requires, then kubelet and friends would stop or restart if the configuration utilities stopped or restarted; the configuration utilities should definitely not have an impact on the runtime of kubelet and friends once started.)

We couldn't find a way to express what we want in systemd, but we should continue research to show that it doesn't actually exist, and then consider our possible approaches:

• augment systemd
• wrap kubelet (and friends) to add domain knowledge
• your idea here?

[tjkirch]

This issue was created before we discovered BindsTo in #143 - using Requires= does not mean that a service will stop/restart if the thing it requires stops/restarts. That's BindsTo's job. I'm going to test whether it just works if we switch these to Requires.

[tjkirch]

Stopping configured.target still stopped the services that Require it. There's a subtlety of Requires I hadn't noticed before; from the docs: "with or without specifying After=, this unit will be stopped if one of the other units is explicitly stopped." We've been testing by explicitly stopping, not through failure. configured.target can't fail, since it's just a target, so maybe that's OK... need a little more thought/research.

[tjkirch]

configured.target can't fail, since it's just a target, so maybe that's OK

There's an option "RefuseManualStop" that we could use to prevent the case of someone stopping configured.target, and thus preventing the cascading stop of things that Require it...

[tjkirch]

I tested "RefuseManualStop" on configured.target, and that worked, in that you could no longer stop it and cause havoc on kubelet, host-containers, and friends.

However, the problem extends all the way up the dependency tree. For example, if you stop settings-applier, it stops configured.target, which stops kubelet etc.

So, we'd have to RefuseManualStop of that whole Requires tree, which would be: configured.target, settings-applier, apiserver, network.target, storewolf, and migrator.

I certainly think we should be able to stop/restart apiserver; the rest are targets or one-shots that probably don't need to be stopped, but RefuseManualStop also blocks restart, and I think you should be able to restart one-shots.

We could make settings-applier Want apiserver instead of Require it, but that's inaccurate and would cause useless applier failures if apiserver couldn't start.

Back to an impasse...

[tjkirch]

And now for something completely different:

We could replace (or augment) configured.target with path units. For example, kubelet could depend on a path unit that's defined with a "PathExists=" check on the thar-be-settings--generated config file. Services would start as soon as config files are generated, and not otherwise. Failure/stop of a configuration-related service later on would not impact the depending service. We could choose whether changes to the file should restart the depending service.

Downsides:

• Duplication of knowledge of the config files
• Timing issues with files existing versus being complete and ready

[arnaldo2792]

I think this was resolved in #1423. With the new systemd boot sequence, kubelet, docker, and the ecs agent won't start if any of the required dependencies (early-boot-config, apiserver, containerd, etc) fails to start. We still use Wants instead of Requires to prevent unwanted reloads, when by some reason either the configured or preconfigured target are restarted. We added RefuseManualStart and RefuseManualStop to all the oneshot services. apiserver can still be restarted without restarting the entire services tree.