Containers with --restart=always policy does not restart services automatically after system reboot.

[amtdas]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Description
Containers with --restart=always policy does not cause services to automatically restart after the system reboot, as it does with Docker.

Steps to reproduce the issue:

1. Create a new container with --restart=always policy

podman run -d --name test-rh83 -p 80:80 --restart=always docker.io/httpd
Trying to pull docker.io/httpd...

podman ps -a
CONTAINER ID  IMAGE                           COMMAND           CREATED             STATUS                 PORTS               NAMES
da1ce8f0ec32  docker.io/library/httpd:latest  httpd-foreground  About a minute ago  Up About a minute ago  0.0.0.0:80->80/tcp  test-rh83

2. Reboot or restart the system/kernel
3. Containers shows in Exited state after system reboot and need automatic start.

podman ps -a
CONTAINER ID  IMAGE                           COMMAND           CREATED         STATUS                         PORTS               NAMES
da1ce8f0ec32  docker.io/library/httpd:latest  httpd-foreground  26 minutes ago  Exited (0) About a minute ago  0.0.0.0:80->80/tcp  test-rh83

Describe the results you received:
Podman don't restart containers after system reboot which were in running state whereas docker does support this after applying --restart=always policy .

Describe the results you expected:
I understand that podman is daemonless and rely completely on systemd for system integration. Having this feature in podman in future release will remove systemd dependency.

Additional information you deem important (e.g. issue happens only occasionally):

podman version
Version:      3.0.2-dev
API Version:  3.0.0
Go Version:   go1.15.7
Built:        Wed Apr  7 18:36:54 2021
OS/Arch:      linux/amd64

podman inspect test-rh83
[
{
"Id": "da1ce8f0ec325a74fe516251c6a5d906fa364e95ad003ff8c1cd51627bb30739",
"Created": "2021-06-01T16:16:00.64145968+10:00",
"Path": "httpd-foreground",
"Args": [
"httpd-foreground"
],
"State": {
"OciVersion": "1.0.1-dev",
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
::::::::::::::::::::::::
},
"RestartPolicy": {
"Name": "always",
"MaximumRetryCount": 0
},

[vrothberg]

Thanks for opening the issue. Since there's already an RFE in the Red Hat Bugzilla [1], I think we should keep the discussion there.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1967421

[mheon]

@vrothberg That bug is private, though? Might be better to keep this open for public discussion.

[vrothberg]

@vrothberg That bug is private, though? Might be better to keep this open for public discussion.

Good catch, thanks, @mheon. Reopened.

[vrothberg]

FWIW, I think we could tackle it via systemd unit that fires off at boot and runs a podman start --filter restartpolicy=always. --filter for podman start doesn't exist (yet) though.

[rhatdan]

Yes I thought this was something @boaz0 was working on, I believe we have all of the parts and now we just need to add a systemd unit file to run at boot time to start the restart=always containers.

[boaz0]

@rhatdan that's correct. This is the PR you're talking about.
@amtdas as @vrothberg mentioned: creating a systemd unit files that runs podman start --filter restart-policy=always --all will resolve your issue.

@amtdas Let me know if it is working for you. If not, I will fix this.
Thanks.

[vrothberg]

@boaz0 @rhatdan, I think we should create a podman-restart.service in https://github.com/containers/podman/tree/master/contrib/systemd/system. This way, it can be installed (and enabled) by packages. What do you think?

[boaz0]

@vrothberg - sounds like a good idea. Can you assign this task to me?

[vrothberg]

@vrothberg - sounds like a good idea. Can you assign this task to me?

Done, thank you!

[amtdas]

Above was mistakenly closed, reopening it.

[nanonyme]

Sorry for commenting on closed issue but is the intent that this goes out with v3.3 or will this be backported? If v3.3, is there any estimate when that will be? This feature is a real quality-of-life enhancer for running Podman server processes.

[mheon]

This is a new feature so it will not be backported. I expect 3.3 to land late this month or early next month.

[plevart]

Sorry for commenting on closed issue, but have you considered to support also the --restart=unless-stopped policy at the same time? The filter should be set-up so that it considers all restart-policy=always containers that are not in RUNNING state (which typically are not at boot time) and also all restart-policy=unless-stopped containers that are not in RUNNING or STOPPED state. When you reboot a machine they typically end up in EXITED state.

[mheon]

We could do so, but we'd need to add a new filter that detects stopped-by-user containers.

[plevart]

We could do so, but we'd need to add a new filter that detects stopped-by-user containers.

Visible with podman ps -a, the container that is stopped by user reports status as Stopped while the container that is not running due to its main process exiting or host reboot is reported with status Exited. So the difference already exists. But I don't know if and how it is expressible in the form of filter.

[mheon]

That's not intended behavior, that's a pretty significant bug - the cleanup process is not running on manually-stopped containers and they may not be unmounted.

[plevart]

What do you mean by "cleanup process"? The equivalent of podman rm ? This should run only on containers started with podman run --rm option...

[mheon]

It can handle --rm, but on containers created without it it takes care of unmounting the container's filesystems, cleaning up its network stack, and generally making sure no resources are being used by containers that are no longer running.

[plevart]

So, you are saying that podman ps -a reporting container status as Stopped is a consequence of this cleanup not being performed while it should be. And if it were performed, podman ps -a would report such container status as Exited being indistinguishable from container that just exited its main process?

[mheon]

Yes.

We do retain an internal state which indicates whether a container was manually stopped by the user (--restart=always needs to know not to restart a container after podman stop) but it is presently not exposed to the user. We could add a filter to expose it, which would allow the systemd unit to filter on it.

[nanonyme]

In case that thing you discussed was a bug, how about creating separate issue about it?

[parmentelat]

for the record, with podman-3.4.7 vanilla-installed on fedora 35, I had to manually enable the systemd service, this is not done automatically by rpm install as I thought at first

systemctl enable podman-restart

[rhatdan]

@lsm5 PTAL, I don't think it is enabled by default even now in podman 4.3. I agree that we should enable this by default.
@lsm5 WDYT?

[ghost]

for the record, with podman-3.4.7 vanilla-installed on fedora 35, I had to manually enable the systemd service, this is not done automatically by rpm install as I thought at first

systemctl enable podman-restart

For rootless containers you may also try (no root/sudo command!): systemctl --user enable podman-restart
System unit did not work for me

[n3gwg]

This solution seems rather interesting, however, I am curios what steps would be needed in pursuance of also making inclusive a "restart-policy=unless-stopped".

For example, sometimes I wish to stop a container and do not wish it to automatically restart. However, if it comes to pass that the Fedora CoreOS instance reboots then yes, I'd surely wish it to automatically restart.

I would suggest simply adding ',restart-policy=unless-stopped' or perhaps instituting some sort of configuration file for the service that could be most easily amended when the need arises.

What say you?

Stuart, N3GWG

[vrothberg]

Thanks for reaching out, @n3gwg! The idea sounds interesting and I understand its value. Would you open a new issue for that? This way, we can better track work on it.

[n3gwg]

Done.

[gloatoriginal]

Hello with podman version 4.3.1 and podman-compose version 1.0.3, my container is not restarting while using:

               "RestartPolicy": {
                    "Name": "unless-stopped",
                    "MaximumRetryCount": 0
               },

I'm currently on Debian 12 Bookworm, I attempted with systemctl --user enable podman-restart and had no success. This is a user ran (non-sudo) container. Thank you.

[vrothberg]

@gloatoriginal, thanks for reaching out. Please create a new issue and provide all the data request in the issue template.

This one here is old and has already been closed. The new issue will help up track things down and triage them individually.

[rhatdan]

Also could you try ths with podman 4.4+ version.

[RevoltPW]

Based on this tutorial I have created a bash script to automate the whole process of Creating a systemd service as non-root user. Make sure to pass CONTAINER_NAME and IMAGE_NAME as arguments..

#!/bin/bash

# Check if the correct number of arguments is provided
if [ "$#" -ne 2 ]; then
    echo "Usage: $0 CONTAINER_NAME IMAGE_NAME"
    exit 1
fi

# Variables
CONTAINER_NAME="$1"
IMAGE_NAME="$2"

# Step 1: Get the container up and running
podman run -d --name "$CONTAINER_NAME" "$IMAGE_NAME"
podman ps

# Step 2: Create a systemd service
# Step 2A: Create a systemd unit file
podman generate systemd --new --name "$CONTAINER_NAME" -f

# Step 2B: Move the systemd service file to a specific location
mkdir -p ~/.config/systemd/user/
mv -v "container-$CONTAINER_NAME.service" ~/.config/systemd/user/

# Step 3: Enable systemd service
systemctl --user daemon-reload
systemctl --user enable "container-$CONTAINER_NAME.service"

# Check the status of the systemd service
systemctl --user status "container-$CONTAINER_NAME.service"

# Stop and remove the container
echo "Stopping and removing the container..."
podman stop "$CONTAINER_NAME"
podman container rm "$CONTAINER_NAME"

# Reboot the system (Uncomment the following line if you want to reboot automatically)
# echo "Rebooting the system in 10 seconds..."
# sleep 10
# sudo reboot

Save the script to a file, for example, podman_systemd_setup.sh, and give it execute permissions using chmod +x podman_systemd_setup.sh. Finally, run the script using ./podman_systemd_setup.sh. This script will create the container, generate a systemd unit file, move it to the appropriate location for a non-root user, and enable the systemd service.

Here's also a simple wrapper script that checks for the --restart always flag, and if present, it runs the provided podman_systemd_setup.sh script after executing the original Podman command:

#!/bin/bash

# Path to the podman_systemd_setup.sh script
SYSTEMD_SETUP_SCRIPT="/path/to/podman_systemd_setup.sh"

# Check if the --restart always flag is passed
if [[ " $* " =~ " --restart always " ]]; then
    RESTART_ALWAYS=1
else
    RESTART_ALWAYS=0
fi

# Run the original Podman command
/usr/bin/podman "$@"

# If the --restart always flag is passed, run the setup script
if [ "$RESTART_ALWAYS" -eq 1 ]; then
    # Extract container name and image name from the arguments
    CONTAINER_NAME="$(echo "$@" | grep -oP '(?<=--name )[^\s]+' | head -n 1)"
    IMAGE_NAME="$(echo "$@" | grep -oP '[^\s]+$')"

    # Run the setup script
    "$SYSTEMD_SETUP_SCRIPT" "$CONTAINER_NAME" "$IMAGE_NAME"
fi

Save this script as podman_wrapper.sh, and give it execute permissions using chmod +x podman_wrapper.sh. Update the SYSTEMD_SETUP_SCRIPT variable with the correct path to your podman_systemd_setup.sh script.

To use this wrapper script instead of the original Podman command, you can create an alias in your shell configuration file (e.g., ~/.bashrc or ~/.zshrc):
alias podman='/path/to/podman_wrapper.sh'

Replace /path/to/podman_wrapper.sh with the actual path to the podman_wrapper.sh script. Then, restart your terminal or run source ~/.bashrc (or source ~/.zshrc) to apply the changes.

Now, when you run a podman run command with the --restart always flag, the wrapper script will call the original Podman command and then execute the podman_systemd_setup.sh script if the flag is present.