Quadlet support depending on health conditions for container

[Lunarequest]

Feature request description

Docker compose has a detective depends_on which can be used to depend on containers in a compose stack and wait to start containers that use this until the health check passes, if they don't the service is not started. An example would be

depends_on:
      db:
        condition: service_healthy

Any container in the compose file with this directive will not start until a container named db starts and has a healthy condition, this is useful for things like postgres which take a bit to start up at which point other services that depend on it may fali as postgres hasn't started up.

Suggest potential solution

I would like a directive under [Container] that mirrors the depends_on field maybe

[Container]
DependsOn=name.container@healthy

where @healthy refers to the condition

Have you considered any alternatives?

I have considered add bash to my container and writing a shell script to wait for the db to come up but I would rather not as my container currently is extremely small and has no shell nor coreutils.

Additional context

No response

[rhatdan]

Wouldn't the systemd way of doing this be just to define a depend relationship between the containers and have the "depended On" container as a notify socket, and wait to notify that the is ready.

[vrothberg]

Wouldn't the systemd way of doing this be just to define a depend relationship between the containers and have the "depended On" container as a notify socket, and wait to notify that the is ready.

I concur. systemd has a very elaborate dependency management system which, for instance, may block starting a unit until a dependency is up and running. It may also restart a unit if one of its dependencies gets restarted.

That being said, I don't think Quadlet should manage these dependencies and let users define the dependencies in the [Service] table (see man systemd.service).

Regarding health checks:

• Podman extended health checks with "custom on-failure" actions which can be configured to kill a container once it gets unhealthy. This in combination with running the container in systemd (e.g., with Quadlet) will yield a restart of the service - if the service's restart policy is set to "on-failure". You can find more details on that in the following article: https://www.redhat.com/sysadmin/podman-edge-healthcheck
• Podman does not yet have means to only send the READY message when the container is healthy. But that's perfectly doable by adding a new --notify=healthcheck policy and busy-wait until the container turns healthy (or the start timeout kicks in).

[vrothberg]

Cc: @ygalblum @alexlarsson

[ygalblum]

In the Quadlet demo, I set the dependency between the WP application .kube and the service created for the mysql .container file in the .kube file: https://github.com/ygalblum/quadlet-demo/blob/main/quadlet-files/quadlet-demo.kube#L5. So, users should be able to set this dependency on their own without needing Quadlet to translate.

Having said that, this means that the naming pattern of the .service files must be guaranteed and cannot be changed in later versions as it will break such dependency

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[Lunarequest]

still important

[rhatdan]

@Lunarequest you never responded to @ygalblum comment.

[Lunarequest]

ah my bad I didn't know i needed to respond. I wanted some way to do a health check specifically as some services eg postgres on init can take a very long time to fully start up which can lead to other services which depend on this service to crash despite it to podman looking like the service has started

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[ygalblum]

@Lunarequest thanks for the added info. I understand the requirement, but I'm not sure that Quadlet is the answer here.

At the end of the day, Quadlet is a systemd generator that links between podman commands (run, kube play) and systemd and does not have a life of its own. So, in order to add such dependency management, we need to understand how it can be achieved by using only systemd and podman.

Systemd can (and should) be used for dependency management making sure that before starting the dependent service the service on which it depends is running. Now comes the question of what does it mean that the service is running.
If it's the mere start of the container, then it's simple (and shown in my demo). But, if we want to make sure that the service has reached a certain state we will need podman to hold off notifying systemd that the service is in fact running until health-startup is successful.
Having said that, assuming I read the code correctly, it seems that podman emits the notification before scheduling the startup health check.

@rhatdan, @vrothberg WDYT?

[ygalblum]

@vrothberg I see that you've just commented on #6160. Isn't this exactly what we are talking about here?

[vrothberg]

@ygalblum, yes, I think so. We need to get #13627 done to make a podman wait --condition=healthy work. This would allow this request and for implementing the --sdnotify=healthy one.

[ygalblum]

@vrothberg since #13627 is done, can we start looking at --sdnotify=healthy?

[vrothberg]

@vrothberg since #13627 is done, can we start looking at --sdnotify=healthy?

Absolutely! I won't find time to work on it short term but feel free to grab it.

[rcasta74]

This request looks similar to what I need, so I'm asking here before to open a potential duplicated issue.

I have initially created a pod manually and I have attached several containers to it; with run/create commands I was able to use --requires option to define dependencies between containers.
When I have later redefined my containers in a yaml file to be used in a .kube systemd unit, I missed this feature.
Could this feature request apply to my use case as well?

[rhatdan]

Does kube yaml have this type of functionality? Init containers?

[rcasta74]

As per my understanding, init containers work differently, so that standard containers won't start until init containers are done.
What I need is to be able to start a container after that another one is already started.

For example when I have created my containers manually, I have created a pod X, than I have created a container A with --pod=X and container B with --pod=X --requires=A. When starting pod X, container B was always started after container A.

My question is whether it is possible to make podman kube play to create containers with option --requires in the same way I can do when I create them manually.

[rcasta74]

Just found out that a new feature for init containers have been recently released on kubernetes side (Sidecar containers).
Is there a plan to support it on podman as well?

[rhatdan]

@ygalblum @umohnani8 PTAL

[nogweii]

I think everything exists in Podman now except for the glue to expose this as a capability inside a .container unit. Looking at the code, quadlet currently treats the Notify= option as a boolean.

I'd like to see it be expanded to a trinary value:

• true/false - keeps their current meaning
• healthy - sets --sdnotify=health which means other systemd units that depend on this one would be blocked until the healthchecks pass

[rhatdan]

Could you open a PR to make that change?

[deuill]

Gave this a go in #20714 since I'm using Quadlet with --sdnotify=healthy quite extensively on my side.