Rootless container doesn't start at systemstart: podman[29817]: Error: no container with name or ID <container ID> found: no such container #8504

topas-rec · 2020-11-28T10:35:05Z

/kind bug

Description

I'm running a rootles container and use podman generate systemd --files --name --restart-policy=always -t 1 181f37a4d457 to generate systemd files. They are placed at /etc/system/systemd/.
The service does not start.
podman[29817]: Error: no container with name or ID frosty_jang found: no such container

When I list all containers the container is present

podman ps -a
CONTAINER ID  IMAGE                               COMMAND               CREATED      STATUS          PORTS                 NAMES
181f37a4d457  docker.io/library/nextcloud:latest  apache2-foregroun...  4 hours ago  Up 2 hours ago  0.0.0.0:8084->80/tcp  frosty_jang

Detailed error log below

Steps to reproduce the issue:

Create a rootless container
Create and copy systemd files to /etc/system/systemd/ to start the container on systemstart
Enable and start the service associated with the container

Describe the results you received:
journalctl output:

Nov 28 12:56:04 tobias-pc sudo[29814]:   tobias : TTY=pts/0 ; PWD=/home/tobias ; USER=root ; COMMAND=/usr/bin/systemctl start container-frosty_jang.service
Nov 28 12:56:04 tobias-pc sudo[29814]: pam_unix(sudo:session): session opened for user root by (uid=0)
Nov 28 12:56:04 tobias-pc systemd[1]: Starting Podman container-frosty_jang.service...
Nov 28 12:56:05 tobias-pc podman[29817]: Error: no container with name or ID frosty_jang found: no such container
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Control process exited, code=exited, status=125/n/a
Nov 28 12:56:05 tobias-pc podman[29850]: Error: no container with name or ID frosty_jang found: no such container
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Control process exited, code=exited, status=125/n/a
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Failed with result 'exit-code'.
Nov 28 12:56:05 tobias-pc systemd[1]: Failed to start Podman container-frosty_jang.service.
Nov 28 12:56:05 tobias-pc sudo[29814]: pam_unix(sudo:session): session closed for user root
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Service RestartSec=100ms expired, scheduling restart.
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Scheduled restart job, restart counter is at 1.
Nov 28 12:56:05 tobias-pc systemd[1]: Stopped Podman container-frosty_jang.service.
Nov 28 12:56:05 tobias-pc systemd[1]: Starting Podman container-frosty_jang.service...
Nov 28 12:56:05 tobias-pc podman[29883]: Error: no container with name or ID frosty_jang found: no such container
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Control process exited, code=exited, status=125/n/a
Nov 28 12:56:05 tobias-pc podman[29921]: Error: no container with name or ID frosty_jang found: no such container
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Control process exited, code=exited, status=125/n/a
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Failed with result 'exit-code'.
Nov 28 12:56:05 tobias-pc systemd[1]: Failed to start Podman container-frosty_jang.service.
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Service RestartSec=100ms expired, scheduling restart.
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Scheduled restart job, restart counter is at 2.
Nov 28 12:56:05 tobias-pc systemd[1]: Stopped Podman container-frosty_jang.service.
Nov 28 12:56:05 tobias-pc systemd[1]: Starting Podman container-frosty_jang.service...
Nov 28 12:56:05 tobias-pc podman[29955]: Error: no container with name or ID frosty_jang found: no such container
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Control process exited, code=exited, status=125/n/a
Nov 28 12:56:05 tobias-pc podman[29988]: Error: no container with name or ID frosty_jang found: no such container
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Control process exited, code=exited, status=125/n/a
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Failed with result 'exit-code'.
Nov 28 12:56:05 tobias-pc systemd[1]: Failed to start Podman container-frosty_jang.service.
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Service RestartSec=100ms expired, scheduling restart.
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Scheduled restart job, restart counter is at 3.
Nov 28 12:56:05 tobias-pc systemd[1]: Stopped Podman container-frosty_jang.service.
Nov 28 12:56:05 tobias-pc systemd[1]: Starting Podman container-frosty_jang.service...
Nov 28 12:56:05 tobias-pc podman[30021]: Error: no container with name or ID frosty_jang found: no such container
Nov 28 12:56:05 tobias-pc systemd[1]: container-frosty_jang.service: Control process exited, code=exited, status=125/n/a
Nov 28 12:56:06 tobias-pc podman[30054]: Error: no container with name or ID frosty_jang found: no such container
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Control process exited, code=exited, status=125/n/a
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Failed with result 'exit-code'.
Nov 28 12:56:06 tobias-pc systemd[1]: Failed to start Podman container-frosty_jang.service.
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Service RestartSec=100ms expired, scheduling restart.
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Scheduled restart job, restart counter is at 4.
Nov 28 12:56:06 tobias-pc systemd[1]: Stopped Podman container-frosty_jang.service.
Nov 28 12:56:06 tobias-pc systemd[1]: Starting Podman container-frosty_jang.service...
Nov 28 12:56:06 tobias-pc podman[30087]: Error: no container with name or ID frosty_jang found: no such container
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Control process exited, code=exited, status=125/n/a
Nov 28 12:56:06 tobias-pc podman[30120]: Error: no container with name or ID frosty_jang found: no such container
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Control process exited, code=exited, status=125/n/a
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Failed with result 'exit-code'.
Nov 28 12:56:06 tobias-pc systemd[1]: Failed to start Podman container-frosty_jang.service.
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Service RestartSec=100ms expired, scheduling restart.
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Scheduled restart job, restart counter is at 5.
Nov 28 12:56:06 tobias-pc systemd[1]: Stopped Podman container-frosty_jang.service.
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Start request repeated too quickly.
Nov 28 12:56:06 tobias-pc systemd[1]: container-frosty_jang.service: Failed with result 'exit-code'.
Nov 28 12:56:06 tobias-pc systemd[1]: Failed to start Podman container-frosty_jang.service.
tobias@tobias-pc:~$

Describe the results you expected:
Service starts without issue

Additional information you deem important (e.g. issue happens only occasionally):
I use gid mapping?

Output of podman version:

Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.14
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.20, commit: '
  cpus: 4
  distribution:
    distribution: debian
    version: "10"
  eventLogger: journald
  hostname: tobias-pc
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.19.0-6-amd64
  linkmode: dynamic
  memFree: 3665915904
  memTotal: 8315002880
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 8387555328
  swapTotal: 8387555328
  uptime: 14m 44.15s
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/tobias/.config/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.4.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  graphRoot: /home/tobias/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/1000/containers
  volumePath: /home/tobias/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14
  OsArch: linux/amd64
  Version: 2.1.1

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 2.1.1~2 amd64 [installed]
podman/unknown 2.1.1~2 arm64
podman/unknown 2.1.1~2 armhf
podman/unknown 2.1.1~2 ppc64el

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Issue #4678 might be related
Searched troubleshooting without finding something related

Additional environment details (AWS, VirtualBox, physical, etc.):

The text was updated successfully, but these errors were encountered:

topas-rec · 2020-11-28T11:49:49Z

In #4678 I don't see the solution and there is no reference to where in the troubleshooting guide the solution is noted.

topas-rec · 2020-11-28T11:59:52Z

I incompletely posted the journalctl output.
I edited the title and the log output in the first comment.

It seems the first issue I'm getting is: podman[29817]: Error: no container with name or ID frosty_jang found: no such container

When I list all containers the container is present

podman ps -a
CONTAINER ID  IMAGE                               COMMAND               CREATED      STATUS          PORTS                 NAMES
181f37a4d457  docker.io/library/nextcloud:latest  apache2-foregroun...  4 hours ago  Up 2 hours ago  0.0.0.0:8084->80/tcp  frosty_jang

Sarting the container without root works (sure, rootless):

tobias@tobias-pc:~$ podman start frosty_jang
frosty_jang

Starting the container as root doesn't work - systemd might start this as root... But rootless containers should work to start on system start with systemd (As mentioned in #4678 for example). Why is this container not found?

tobias@tobias-pc:~$ sudo podman stop frosty_jang
Error: no container with name or ID frosty_jang found: no such container

Luap99 · 2020-11-28T12:45:53Z

Rootless containers are not supposed to be run as root. You have to start them in your systemd user session.
Please see this how to install the service files as user: https://github.com/containers/podman/blob/master/docs/source/markdown/podman-generate-systemd.1.md#installation-of-generated-systemd-unit-files

topas-rec · 2020-11-28T14:19:49Z

Thanks for helping me.

I found this in https://github.com/containers/podman/blob/master/docs/source/markdown/podman-generate-systemd.1.md#installation-of-generated-systemd-unit-files

Note: Coping unit files to /etc/systemd/system and enabling it marks the unit file to be automatically started at boot. And smillarly, coping a unit file to $HOME/.config/systemd/user and enabling it marks the unit file to be automatically started on user login.

I planned to run a container at boot, therefore I copied the systemd files to /etc/systemd/system.

Can I start rootless containers at boot without a user logged in? And how do I do that?

Luap99 · 2020-11-28T14:38:08Z

If you run loginctl enable-linger <username> and have the unit enabled it should start at boot.

mheon · 2020-11-28T14:46:45Z

You will also need to set User to the user who should be running the container. Please note, though, that this is something the systemd team doesn't really support that well, the intention there is to run the units on the user that will launch the service.

…

On Sat, Nov 28, 2020, 09:38 Luap99 ***@***.***> wrote: If you run loginctl enable-linger <username> and have the unit enabled it should start at boot. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8504 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCBPQXSKXROSUFXNESLSSEDN7ANCNFSM4UFWWHZA> .

topas-rec · 2020-11-28T20:05:44Z

Thanks I'll keep that in mind.

Currently I use the user systemd.
Here the start command does not return. I have to stop it (CTRL-C). The container starts, but the status of the service is activating

The errors in journalctl are not as verbose as the ones in /var/log/syslog and therefore I'm posting them here:

Nov 28 21:01:24 tobias-pc systemd[5225]: Starting Podman container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service...
Nov 28 21:01:25 tobias-pc podman[21425]: 2020-11-28 21:01:25.219565372 +0100 CET m=+0.327012366 container init 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65 (image=docker.io/library/nextcloud:latest, name=frosty_jang)
Nov 28 21:01:25 tobias-pc podman[21425]: 2020-11-28 21:01:25.234144024 +0100 CET m=+0.341590961 container start 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65 (image=docker.io/library/nextcloud:latest, name=frosty_jang)
Nov 28 21:01:25 tobias-pc podman[21425]: 181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65
Nov 28 21:01:25 tobias-pc systemd[5225]: container-181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65.service: Can't open PID file /run/user/1000/containers/overlay-containers/181f37a4d4577fbcf1e7fc2cca6699b7a2906ec3b34134b43404542bda2ffa65/userdata/conmon.pid (yet?) after start: Permission denied

Stopping the service (and the container) works via systemd --user and the stop-process returns.
Does someone know why the PID file cannot be opened?

topas-rec · 2020-11-28T20:34:57Z

Created #8506 for the remaining issue for better tracking.

mheon · 2020-11-28T20:35:23Z

Ah, I forgot about that issue. Systemd is complaining about the ownership of the PID file - systemd, when run as root, wants all PID files to also be owned by root (for security reasons). This protection even applies to unit files that use the User directive, which are clearly not running as root (so you'd imagine they'd allow the user that ran the process to be the owner of the PID file?). We talked with the systemd team about this and they felt that their current behavior is correct.

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Nov 28, 2020

topas-rec changed the title ~~Rootless container doesn't start at systemstart: Service RestartSec=100ms expired, scheduling restart~~ Rootless container doesn't start at systemstart: podman[29817]: Error: no container with name or ID <container ID> found: no such container Nov 28, 2020

topas-rec closed this as completed Nov 28, 2020

topas-rec mentioned this issue Nov 28, 2020

Start of rootless container in systemd hangs (Can't open PID file /run/user/1000/containers/overlay-containers/[...].pid (yet?) after start: Permission denied) #8506

Closed

gczarnocki mentioned this issue Jan 3, 2022

podman generate systemd with added User= directive for rootless container fails with PID file is not owned by root. error #12736

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rootless container doesn't start at systemstart: podman[29817]: Error: no container with name or ID <container ID> found: no such container #8504

Rootless container doesn't start at systemstart: podman[29817]: Error: no container with name or ID <container ID> found: no such container #8504

topas-rec commented Nov 28, 2020 •

edited

Loading

topas-rec commented Nov 28, 2020

topas-rec commented Nov 28, 2020 •

edited

Loading

Luap99 commented Nov 28, 2020

topas-rec commented Nov 28, 2020

Luap99 commented Nov 28, 2020

mheon commented Nov 28, 2020 via email

topas-rec commented Nov 28, 2020 •

edited

Loading

topas-rec commented Nov 28, 2020

mheon commented Nov 28, 2020

Rootless container doesn't start at systemstart: podman[29817]: Error: no container with name or ID <container ID> found: no such container #8504

Rootless container doesn't start at systemstart: podman[29817]: Error: no container with name or ID <container ID> found: no such container #8504

Comments

topas-rec commented Nov 28, 2020 • edited Loading

topas-rec commented Nov 28, 2020

topas-rec commented Nov 28, 2020 • edited Loading

Luap99 commented Nov 28, 2020

topas-rec commented Nov 28, 2020

Luap99 commented Nov 28, 2020

mheon commented Nov 28, 2020 via email

topas-rec commented Nov 28, 2020 • edited Loading

topas-rec commented Nov 28, 2020

mheon commented Nov 28, 2020

topas-rec commented Nov 28, 2020 •

edited

Loading

topas-rec commented Nov 28, 2020 •

edited

Loading

topas-rec commented Nov 28, 2020 •

edited

Loading