Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman generate systemd service file fails to start container #11264

Closed
cryobry opened this issue Aug 18, 2021 · 2 comments
Closed

podman generate systemd service file fails to start container #11264

cryobry opened this issue Aug 18, 2021 · 2 comments
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@cryobry
Copy link

cryobry commented Aug 18, 2021 •
edited
Loading

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Note that this problem is only happening on my workstation, it does not reproduce in my test VMs.

My containers start fine using podman directly but when I attempt to start them via a podman generate systemd generated systemd user service file, they will not start. Using the default service file initially results in a pidfile error (see below) and after commenting out the pidfile= check, the service dies without successfully launching the container.

Steps to reproduce the issue:

  1. Use podman generate systemd --name sabnzbd --restart-policy no --container-prefix "" --separator "" > ~/.config/systemd/user/sabnzbd.service to create the service file:
# /home/bryan/.config/systemd/user/sabnzbd.service
# sabnzbd.service
# autogenerated by Podman 3.3.0-rc2
# Wed Aug 18 12:08:42 EDT 2021

[Unit]
Description=Podman sabnzbd.service
Documentation=man:podman-generate-systemd(1)
Wants=network-online.target
After=network-online.target
RequiresMountsFor=/run/user/1000

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=no
TimeoutStopSec=70
ExecStart=/usr/bin/podman start sabnzbd
ExecStop=/usr/bin/podman stop -t 10 sabnzbd
ExecStopPost=/usr/bin/podman stop -t 10 sabnzbd
PIDFile=/run/user/1000/overlay-containers/a74e21255bef9f2162a07287231f56e5737ece7328ceb71dc91c95f5a114942f/userdata/conmon.pid
Type=forking

[Install]
WantedBy=multi-user.target default.target
  1. systemctl --user daemon-reload && systemctl --user start sabnzbd

Describe the results you received:

Problem 1:

× sabnzbd.service - Podman sabnzbd.service
     Loaded: loaded (/home/bryan/.config/systemd/user/sabnzbd.service; enabled; vendor preset: disabled)
     Active: failed (Result: protocol) since Wed 2021-08-18 12:08:53 EDT; 19ms ago
       Docs: man:podman-generate-systemd(1)
    Process: 930290 ExecStart=/usr/bin/podman start sabnzbd (code=exited, status=0/SUCCESS)
    Process: 930392 ExecStopPost=/usr/bin/podman stop -t 10 sabnzbd (code=exited, status=0/SUCCESS)
        CPU: 83ms

Aug 18 12:08:42 workstation systemd[1945]: Starting Podman sabnzbd.service...
Aug 18 12:08:42 workstation systemd[1945]: sabnzbd.service: New main PID 930383 does not belong to service, and PID file is not owned by root. Refusing.
Aug 18 12:08:42 workstation systemd[1945]: sabnzbd.service: New main PID 930383 does not belong to service, and PID file is not owned by root. Refusing.
Aug 18 12:08:53 workstation systemd[1945]: sabnzbd.service: Failed with result 'protocol'.
Aug 18 12:08:53 workstation systemd[1945]: Failed to start Podman sabnzbd.service.

Problem 2:
If I comment out the PIDFile= line, attempting to start the container via the systemd service results in an inactive (dead) unit:

○ sabnzbd.service - Podman sabnzbd.service
     Loaded: loaded (/home/bryan/.config/systemd/user/sabnzbd.service; enabled; vendor preset: disabled)
     Active: inactive (dead) since Wed 2021-08-18 12:11:03 EDT; 6min ago
       Docs: man:podman-generate-systemd(1)
    Process: 931270 ExecStart=/usr/bin/podman start sabnzbd (code=exited, status=0/SUCCESS)
    Process: 931323 ExecStop=/usr/bin/podman stop -t 10 sabnzbd (code=exited, status=0/SUCCESS)
    Process: 931493 ExecStopPost=/usr/bin/podman stop -t 10 sabnzbd (code=exited, status=0/SUCCESS)
        CPU: 133ms

Aug 18 12:10:53 workstation systemd[1945]: Starting Podman sabnzbd.service...
Aug 18 12:11:03 workstation systemd[1945]: Started Podman sabnzbd.service.

Describe the results you expected:

The service file works fine in my test VMs, giving the folliowing output:

● sabnzbd.service - Podman container-sabnzbd.service
     Loaded: loaded (/home/bryan/.config/systemd/user/sabnzbd.service; enabled; vendor preset: disabled)
     Active: active (running) since Tue 2021-08-17 22:27:26 EDT; 16s ago
       Docs: man:podman-generate-systemd(1)
    Process: 20599 ExecStart=/usr/bin/podman start sabnzbd (code=exited, status=0/SUCCESS)
   Main PID: 20650 (conmon)
      Tasks: 27 (limit: 4700)
     Memory: 36.1M
        CPU: 522ms
     CGroup: /user.slice/user-1000.slice/[email protected]/app.slice/sabnzbd.service
             ├─20614 /usr/bin/fuse-overlayfs -o ,lowerdir=/home/bryan/.local/share/containers/storage/overlay/l/JZUF6CC7NDUMSI2RJ62MAPR26P,upperdir=/home/bryan/.local/share/containers/storage/overlay/5f989776b07eb398bfa1f5f6d1c8f64242664bc1e35e04a264abdda95711973c/diff,workdir=/home/bryan/.local/share/containers/storage/overlay/5f989776b07eb398bfa1f5f6d1c8f64242664bc1e35e04a264abdda95711973c/work,context="system_u:object_r:container_file_t:s0:c49,c640" /home/bryan/.local/share/containers/storage/overlay/5f989776b07eb398bfa1f5f6d1c8f64242664bc1e35e04a264abdda95711973c/merged
             ├─20619 /usr/bin/slirp4netns --disable-host-loopback --mtu=65520 --enable-sandbox --enable-seccomp -c -e 3 -r 4 --netns-type=path /run/user/1000/netns/cni-973fd424-8528-83e7-0927-d0ba0b4fd249 tap0
             ├─20621 containers-rootlessport
             ├─20632 containers-rootlessport-child
             ├─20642 /usr/bin/conmon --api-version 1 -c 3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023 -u 3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023 -r /usr/bin/crun -b /home/bryan/.local/share/containers/storage/overlay-containers/3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023/userdata -p /run/user/1000/containers/overlay-containers/3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023/userdata/pidfile -n d5d4c2a63b64-infra --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l k8s-file:/home/bryan/.local/share/containers/storage/overlay-containers/3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023/userdata/ctr.log --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1000/containers/overlay-containers/3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023/userdata/oci-log --conmon-pidfile /run/user/1000/containers/overlay-containers/3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/bryan/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023
             ├─20648 /usr/bin/fuse-overlayfs -o ,lowerdir=/home/bryan/.local/share/containers/storage/overlay/l/KLEJBAC5KCCE74P4DMKAONWQUP:/home/bryan/.local/share/containers/storage/overlay/l/BPZLPNSK4E5SMRDTADK7RZJBTL,upperdir=/home/bryan/.local/share/containers/storage/overlay/f27bb6ee427c299b953616e3fa4d3f9b97d25b05ad74b96c400522ee8f9f6347/diff,workdir=/home/bryan/.local/share/containers/storage/overlay/f27bb6ee427c299b953616e3fa4d3f9b97d25b05ad74b96c400522ee8f9f6347/work,context="system_u:object_r:container_file_t:s0:c49,c640" /home/bryan/.local/share/containers/storage/overlay/f27bb6ee427c299b953616e3fa4d3f9b97d25b05ad74b96c400522ee8f9f6347/merged
             └─20650 /usr/bin/conmon --api-version 1 -c 3eda844c196f4ebcedd325bf5492168d603c4dfa9878742c9969261366c94abe -u 3eda844c196f4ebcedd325bf5492168d603c4dfa9878742c9969261366c94abe -r /usr/bin/crun -b /home/bryan/.local/share/containers/storage/overlay-containers/3eda844c196f4ebcedd325bf5492168d603c4dfa9878742c9969261366c94abe/userdata -p /run/user/1000/containers/overlay-containers/3eda844c196f4ebcedd325bf5492168d603c4dfa9878742c9969261366c94abe/userdata/pidfile -n sabnzbd --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l k8s-file:/home/bryan/.local/share/containers/storage/overlay-containers/3eda844c196f4ebcedd325bf5492168d603c4dfa9878742c9969261366c94abe/userdata/ctr.log --log-level warning --runtime-arg --log-format=json --runtime-arg --log --runtime-arg=/run/user/1000/containers/overlay-containers/3eda844c196f4ebcedd325bf5492168d603c4dfa9878742c9969261366c94abe/userdata/oci-log --conmon-pidfile /run/user/1000/containers/overlay-containers/3eda844c196f4ebcedd325bf5492168d603c4dfa9878742c9969261366c94abe/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/bryan/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg warning --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 3eda844c196f4ebcedd325bf5492168d603c4dfa9878742c9969261366c94abe

Aug 17 22:27:25 fedora systemd[962]: Starting Podman container-sabnzbd.service...
Aug 17 22:27:25 fedora podman[20599]: 2021-08-17 22:27:25.567707692 -0400 EDT m=+0.083433221 container restart 3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023 (image=k8s.gcr.io/pause:3.5, name=d5d4c2a63b64-infra)
Aug 17 22:27:25 fedora podman[20599]: 2021-08-17 22:27:25.783105539 -0400 EDT m=+0.298831024 container init 3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023 (image=k8s.gcr.io/pause:3.5, name=d5d4c2a63b64-infra)
Aug 17 22:27:25 fedora podman[20599]: 2021-08-17 22:27:25.795392477 -0400 EDT m=+0.311117962 container start 3d69291b00d73a88cc4409b7f816a8a7dec88a178e8a77f8ba9bb361c55a0023 (image=k8s.gcr.io/pause:3.5, name=d5d4c2a63b64-infra)
Aug 17 22:27:25 fedora podman[20599]: 2021-08-17 22:27:25.861285383 -0400 EDT m=+0.377010869 container init 3eda844c196f4ebcedd325bf5492168d603c4dfa9878742c9969261366c94abe (image=localhost/sabnzbd:latest, name=sabnzbd, io.buildah.version=1.22.0, maintainer=Bryan Roessler <[email protected]>)
Aug 17 22:27:25 fedora podman[20599]: 2021-08-17 22:27:25.874996077 -0400 EDT m=+0.390721573 container start 3eda844c196f4ebcedd325bf5492168d603c4dfa9878742c9969261366c94abe (image=localhost/sabnzbd:latest, name=sabnzbd, io.buildah.version=1.22.0, maintainer=Bryan Roessler <[email protected]>)
Aug 17 22:27:25 fedora podman[20599]: sabnzbd
Aug 17 22:27:26 fedora systemd[962]: Started Podman container-sabnzbd.service.

Additional information you deem important (e.g. issue happens only occasionally):

Only happening on my workstation.

Output of podman version:

Version:      3.3.0-rc2
API Version:  3.3.0-rc2
Go Version:   go1.16.6
Built:        Mon Aug 16 09:31:20 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

podman info --debug
panic: runtime error: index out of range [1] with length 1

goroutine 1 [running]:
panic(0x558f52d412e0, 0xc00037c1c8)
        /usr/lib/golang/src/runtime/panic.go:1065 +0x565 fp=0xc0003d32f0 sp=0xc0003d3228 pc=0x558f514299e5
runtime.goPanicIndex(0x1, 0x1)
        /usr/lib/golang/src/runtime/panic.go:88 +0xa5 fp=0xc0003d3338 sp=0xc0003d32f0 pc=0x558f51426da5
github.com/containers/podman/pkg/cgroups.getCgroupPathForCurrentProcess(0x0, 0x0, 0x0, 0x0)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/pkg/cgroups/cgroups.go:234 +0x365 fp=0xc0003d3448 sp=0xc0003d3338 pc=0x558f51be26a5
github.com/containers/podman/pkg/cgroups.getAvailableControllers(0x0, 0x1, 0x0, 0xc0003d35a0, 0x558f524ae2c5, 0x558f5280cfe2, 0x22)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/pkg/cgroups/cgroups.go:135 +0x30f fp=0xc0003d3530 sp=0xc0003d3448 pc=0x558f51be196f
github.com/containers/podman/pkg/cgroups.GetAvailableControllers(0x0, 0x1, 0x0, 0x0, 0x2, 0x558f5143e9b7, 0x8)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/pkg/cgroups/cgroups.go:184 +0x45 fp=0xc0003d35b0 sp=0xc0003d3530 pc=0x558f51be1da5
github.com/containers/podman/libpod.(*Runtime).hostInfo(0xc0004c9a40, 0x9, 0xc000312190, 0x9)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/libpod/info.go:105 +0x174 fp=0xc0003d39f8 sp=0xc0003d35b0 pc=0x558f52445fb4
github.com/containers/podman/libpod.(*Runtime).info(0xc0004c9a40, 0x0, 0x0, 0x1)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/libpod/info.go:39 +0x16d fp=0xc0003d3c28 sp=0xc0003d39f8 pc=0x558f5244580d
github.com/containers/podman/libpod.(*Runtime).Info(...)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/libpod/runtime.go:875
github.com/containers/podman/pkg/domain/infra/abi.(*ContainerEngine).Info(0xc0006ba308, 0x558f52e9ef48, 0xc00058f710, 0x558f51a7682a, 0xc0002afb00, 0xc0003d3d08)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/pkg/domain/infra/abi/system.go:31 +0x34 fp=0xc0003d3c98 sp=0xc0003d3c28 pc=0x558f5261fdf4
github.com/containers/podman/cmd/podman/system.info(0x558f539cdb60, 0xc000123810, 0x0, 0x1, 0x0, 0x0)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/cmd/podman/system/info.go:72 +0x78 fp=0xc0003d3d40 sp=0xc0003d3c98 pc=0x558f527a45f8
github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).execute(0x558f539cdb60, 0xc00003c1a0, 0x1, 0x1, 0x558f539cdb60, 0xc00003c1a0)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:856 +0x472 fp=0xc0003d3e00 sp=0xc0003d3d40 pc=0x558f51a755b2
github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x558f539d0ae0, 0xc0000480b8, 0x558f52c0c900, 0x558f53a92210)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:974 +0x375 fp=0xc0003d3ee0 sp=0xc0003d3e00 pc=0x558f51a761d5
github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:902
github.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:895
main.Execute()
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/cmd/podman/root.go:90 +0xe9 fp=0xc0003d3f48 sp=0xc0003d3ee0 pc=0x558f527c1489
main.main()
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/cmd/podman/main.go:39 +0x94 fp=0xc0003d3f88 sp=0xc0003d3f48 pc=0x558f527c0d94
runtime.main()
        /usr/lib/golang/src/runtime/proc.go:225 +0x263 fp=0xc0003d3fe0 sp=0xc0003d3f88 pc=0x558f5142c663
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0003d3fe8 sp=0xc0003d3fe0 pc=0x558f51462721

goroutine 2 [force gc (idle)]:
runtime.gopark(0x558f52e404c0, 0x558f53a5b010, 0x1411, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00007afb0 sp=0xc00007af90 pc=0x558f5142caa6
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:342
runtime.forcegchelper()
        /usr/lib/golang/src/runtime/proc.go:276 +0xc5 fp=0xc00007afe0 sp=0xc00007afb0 pc=0x558f5142c905
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00007afe8 sp=0xc00007afe0 pc=0x558f51462721
created by runtime.init.7
        /usr/lib/golang/src/runtime/proc.go:264 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x558f52e404c0, 0x558f53a5d860, 0x140c, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00007b7a8 sp=0xc00007b788 pc=0x558f5142caa6
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:342
runtime.bgsweep(0xc0000a2000)
        /usr/lib/golang/src/runtime/mgcsweep.go:182 +0x145 fp=0xc00007b7d8 sp=0xc00007b7a8 pc=0x558f51417845
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00007b7e0 sp=0xc00007b7d8 pc=0x558f51462721
created by runtime.gcenable
        /usr/lib/golang/src/runtime/mgc.go:217 +0x5e

goroutine 4 [sleep]:
runtime.gopark(0x558f52e404c0, 0x558f53a5e340, 0x1313, 0x2)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00007bf20 sp=0xc00007bf00 pc=0x558f5142caa6
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:342
runtime.scavengeSleep(0x5ca6d, 0xe0640)
        /usr/lib/golang/src/runtime/mgcscavenge.go:241 +0xc5 fp=0xc00007bf78 sp=0xc00007bf20 pc=0x558f514156e5
runtime.bgscavenge(0xc0000a2000)
        /usr/lib/golang/src/runtime/mgcscavenge.go:366 +0x1ed fp=0xc00007bfd8 sp=0xc00007bf78 pc=0x558f5141590d
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00007bfe0 sp=0xc00007bfd8 pc=0x558f51462721
created by runtime.gcenable
        /usr/lib/golang/src/runtime/mgc.go:218 +0x85

goroutine 5 [finalizer wait]:
runtime.gopark(0x558f52e404c0, 0x558f53a92120, 0xc000111410, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00007a758 sp=0xc00007a738 pc=0x558f5142caa6
runtime.goparkunlock(...)
        /usr/lib/golang/src/runtime/proc.go:342
runtime.runfinq()
        /usr/lib/golang/src/runtime/mfinal.go:175 +0xab fp=0xc00007a7e0 sp=0xc00007a758 pc=0x558f5140c8ab
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00007a7e8 sp=0xc00007a7e0 pc=0x558f51462721
created by runtime.createfing
        /usr/lib/golang/src/runtime/mfinal.go:156 +0x66

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc0004168c0, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00007c760 sp=0xc00007c740 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00007c7e0 sp=0xc00007c760 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00007c7e8 sp=0xc00007c7e0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc000116000, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00007cf60 sp=0xc00007cf40 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00007cfe0 sp=0xc00007cf60 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00007cfe8 sp=0xc00007cfe0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 18 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc000580000, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000076760 sp=0xc000076740 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0000767e0 sp=0xc000076760 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000767e8 sp=0xc0000767e0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc000116020, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000588760 sp=0xc000588740 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0005887e0 sp=0xc000588760 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0005887e8 sp=0xc0005887e0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc000116040, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000076f60 sp=0xc000076f40 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc000076fe0 sp=0xc000076f60 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000076fe8 sp=0xc000076fe0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 20 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc000580020, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000077760 sp=0xc000077740 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0000777e0 sp=0xc000077760 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000777e8 sp=0xc0000777e0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc0004168e0, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000588f60 sp=0xc000588f40 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc000588fe0 sp=0xc000588f60 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000588fe8 sp=0xc000588fe0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc000116060, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00007d760 sp=0xc00007d740 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00007d7e0 sp=0xc00007d760 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00007d7e8 sp=0xc00007d7e0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 9 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc000416900, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00007df60 sp=0xc00007df40 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc00007dfe0 sp=0xc00007df60 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00007dfe8 sp=0xc00007dfe0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 10 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc000416920, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000584760 sp=0xc000584740 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0005847e0 sp=0xc000584760 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0005847e8 sp=0xc0005847e0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 11 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc000416940, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000584f60 sp=0xc000584f40 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc000584fe0 sp=0xc000584f60 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc000584fe8 sp=0xc000584fe0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 12 [GC worker (idle)]:
runtime.gopark(0x558f52e40358, 0xc000416960, 0x1418, 0x0)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000585760 sp=0xc000585740 pc=0x558f5142caa6
runtime.gcBgMarkWorker()
        /usr/lib/golang/src/runtime/mgc.go:1911 +0x105 fp=0xc0005857e0 sp=0xc000585760 pc=0x558f51410625
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0005857e8 sp=0xc0005857e0 pc=0x558f51462721
created by runtime.gcBgMarkStartWorkers
        /usr/lib/golang/src/runtime/mgc.go:1835 +0x3a

goroutine 36 [chan receive]:
runtime.gopark(0x558f52e402b8, 0xc00011c058, 0xc00010170e, 0x2)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00058aed0 sp=0xc00058aeb0 pc=0x558f5142caa6
runtime.chanrecv(0xc00011c000, 0xc00058afb0, 0xc000120001, 0xc00011c000)
        /usr/lib/golang/src/runtime/chan.go:576 +0x37a fp=0xc00058af60 sp=0xc00058aed0 pc=0x558f513f907a
runtime.chanrecv2(0xc00011c000, 0xc00058afb0, 0x558f518ecfcb)
        /usr/lib/golang/src/runtime/chan.go:444 +0x2b fp=0xc00058af90 sp=0xc00058af60 pc=0x558f513f8ceb
github.com/containers/podman/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x558f53a606c0)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/vendor/k8s.io/klog/v2/klog.go:1164 +0x8d fp=0xc00058afd8 sp=0xc00058af90 pc=0x558f520b842d
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00058afe0 sp=0xc00058afd8 pc=0x558f51462721
created by github.com/containers/podman/vendor/k8s.io/klog/v2.init.0
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/vendor/k8s.io/klog/v2/klog.go:418 +0xdf

goroutine 15 [select, locked to thread]:
runtime.gopark(0x558f52e40510, 0x0, 0x1809, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00058a610 sp=0xc00058a5f0 pc=0x558f5142caa6
runtime.selectgo(0xc00058a7a8, 0xc00058a7a0, 0x0, 0x0, 0x2, 0x1, 0x1, 0x1)
        /usr/lib/golang/src/runtime/select.go:327 +0xef7 fp=0xc00058a748 sp=0xc00058a610 pc=0x558f5143de77
runtime.ensureSigM.func1()
        /usr/lib/golang/src/runtime/signal_unix.go:889 +0x1cd fp=0xc00058a7e0 sp=0xc00058a748 pc=0x558f5145a86d
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00058a7e8 sp=0xc00058a7e0 pc=0x558f51462721
created by runtime.ensureSigM
        /usr/lib/golang/src/runtime/signal_unix.go:872 +0xd7

goroutine 16 [syscall]:
runtime.notetsleepg(0x558f53a92e80, 0xffffffffffffffff, 0x0)
        /usr/lib/golang/src/runtime/lock_futex.go:235 +0x38 fp=0xc00058b798 sp=0xc00058b768 pc=0x558f513fef98
os/signal.signal_recv(0x0)
        /usr/lib/golang/src/runtime/sigqueue.go:168 +0xa5 fp=0xc00058b7c0 sp=0xc00058b798 pc=0x558f5145ee25
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:23 +0x25 fp=0xc00058b7e0 sp=0xc00058b7c0 pc=0x558f51b877c5
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00058b7e8 sp=0xc00058b7e0 pc=0x558f51462721
created by os/signal.Notify.func1.1
        /usr/lib/golang/src/os/signal/signal.go:151 +0x46

goroutine 50 [select]:
runtime.gopark(0x558f52e40510, 0x0, 0x1809, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc00061fd80 sp=0xc00061fd60 pc=0x558f5142caa6
runtime.selectgo(0xc00061ffb0, 0xc00061ff08, 0x0, 0x0, 0x2, 0x1, 0x0, 0x0)
        /usr/lib/golang/src/runtime/select.go:327 +0xef7 fp=0xc00061feb8 sp=0xc00061fd80 pc=0x558f5143de77
github.com/containers/podman/libpod/shutdown.Start.func1()
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:45 +0xab fp=0xc00061ffe0 sp=0xc00061feb8 pc=0x558f523bad6b
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00061ffe8 sp=0xc00061ffe0 pc=0x558f51462721
created by github.com/containers/podman/libpod/shutdown.Start
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:44 +0x116

goroutine 52 [sleep]:
runtime.gopark(0x558f52e404f8, 0xc0001201e0, 0xc000701313, 0x1)
        /usr/lib/golang/src/runtime/proc.go:336 +0xe6 fp=0xc000078610 sp=0xc0000785f0 pc=0x558f5142caa6
time.Sleep(0x5f5e100)
        /usr/lib/golang/src/runtime/time.go:193 +0xd8 fp=0xc000078650 sp=0xc000078610 pc=0x558f5145f7d8
github.com/containers/podman/libpod.(*Runtime).libimageEvents.func2(0xc00029ac60, 0x558f52e3ba18, 0xc0004c9a40)
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/libpod/runtime.go:744 +0x25f fp=0xc0000787c8 sp=0xc000078650 pc=0x558f524ddaff
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc0000787d0 sp=0xc0000787c8 pc=0x558f51462721
created by github.com/containers/podman/libpod.(*Runtime).libimageEvents
        /builddir/build/BUILD/podman-3.3.0-rc2/_build/src/github.com/containers/podman/libpod/runtime.go:721 +0x99
zsh: IOT instruction (core dumped)  podman info --debug

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.3.0-0.13.rc2.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

I have tried using podman-3.2.x and podman podman-3.3.0-0.13.rc2

Additional environment details (AWS, VirtualBox, physical, etc.):
@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Aug 18, 2021
@cryobry
Copy link
Author

cryobry commented Aug 18, 2021

Due to the errors from podman info --debug I attempted to remove all of my containers and images, uninstall podman completely, erase all of my container storage in ~/.local/containers, runtime data from /run/user/1000/containers/* and reinstall podman, however this did not fix the problem.

I ended up running a podman system reset and now my systemd services work fine and podman info --debug no longer crashes. I'm not sure what was going on but podman system reset was a satisfactory solution in my case.

@cryobry cryobry closed this as completed Aug 18, 2021
@vrothberg
Copy link
Member

Glad you could resolve the issue, @cryobry. Thanks for mentioning how you resolved it.

That's quite curious. So far I have only seen systemd rejecting the PIDFILE if a system service set User=rootless.

A wild guess: maybe the rootless storage was created in a way that made systemd reject reading the file? (Cc @msekletar)

Note that starting with Podman v3.3.0, units created via podman generate systemd **--new** will use type=notify.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@vrothberg @cryobry