[Bug]: podman pod create command hangs indefinitely #17224
Labels
kind/bug
Categorizes issue or PR as related to a bug.
locked - please file new issue/PR
Assist humans wanting to comment on an old issue or PR with locked comments.
Comments
|
how long does it take if you stop the current running pod first If that is still slow, you may want to shorten the time it waits before killing the containers: |
|
Actually, there is no pod with that name running according to |
|
Has the network, name, replace or userns option actually anything to do with it? I would helpful if you could remove the unnecessary options from the reproducer command. If the hang happens could you check for all running podman commands including the child commands ( |
|
When I remove other options: Both |
|
I don't see any podman command in your ps output. You need to execute it while the podman command hangs. Also you can send the |
|
I did execute it while the Below is the stacktrace. |
|
It looks like the pod create command exited and now the lock is locked forever, that should never happen. Did you reboot? That should fix the lock, but it is still interesting how exactly this can be reproduced. Can you send the signal to the |
|
It hangs even after a reboot, I've tried several times. Here's the output when I run |
|
Does @mheon any ideas? |
|
Something like this could happen if the pod and its infra container were assigned the same lock number, but that shouldn't be possible during Any chance this is using a non-glibc C library (musl, maybe)? I'm not that familiar with Fedora IoT. |
|
Running |
|
OK. Probably a lock conflict, then. We added some detection around that ( |
|
A friendly reminder that this issue had no activity for 30 days. |
|
Without a reproducer on how to get this to happen I do not see how to debug it. In theory it should be impossible for this to happen. |
|
can you provide more info on: from |
|
@giuseppe Of course. I'm on a Btrfs filesystem, though I run the majority of my containers off of an NFS-mounted volume called This is using NFS 4. The pod that had issues ran containers with volumes located on this |
|
thanks. So the NFS mount is used only for volumes |
|
A friendly reminder that this issue had no activity for 30 days. |
|
@giuseppe any other ideas? |
|
Hi there, I ran into this very same issue on a production system and it bugged me for weeks. I could create & run some containers but not others. And once a name was used and hit the lock bug any podman command involving this container name (inspect or ps) would hang forever./home/pascal2/Travail/Sysadmin/Podman The See the attached file for podman info details but I see some similarity with the others that were hit :
Beside that I run podman 3.4.4 on Ubuntu 22.04 |
that podman version is too old, please try something newer |
|
This is the official stable version in Ubuntu repository. |
|
Unfortunately, that does not change the fact that 3.x releases are too old to receive upstream support. We do provide a Kubic repo which includes more recent Podman packages for Ubuntu, with installation instructions available on https://podman.io/docs/installation#ubuntu Podman 4.6 includes a dedicated command for debugging locking issues, |
|
I sure know about and use Kubic repos on other systems, with more or less troubles. Anyway thanks for the hint about the podman locks command. |
|
4.4.1 is currently released in RHEL, and 4.6.1 will be released into RHEL in the coming months, so yes, we are very confident in our current releases. |
|
Ok, I will plan the switch to 4.6 then. Thanks for your answers and good work ! :) |
|
Should this issue be closed? |
Seems that this needs still some more work to prevent this issue to arise. |
|
Hi, I'm facing same issue. I had 4.2.0 so I updated to latest (4.4.1). running rhel 8. any podman pod-related commands hangs. |
|
Please open a new issue and fill out the full issue template |
stale-locking-app
bot
added
the
locked - please file new issue/PR
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Issue Description
podman pod createcommand hangs indefinitely and causes all other podman commands to hang.Steps to reproduce the issue
Describe the results you received
The
podman pod createcommand hangs indefinitely and any podman commands such aspodman pshang while thepodman pod createcommand is hanging.Describe the results you expected
I expected
podman pod createto finish executing almost immediately but within at most a few minutes.podman info output
host: arch: arm64 buildahVersion: 1.28.0 cgroupControllers: - memory - pids cgroupManager: systemd cgroupVersion: v2 conmon: package: conmon-2.1.5-1.fc37.aarch64 path: /usr/bin/conmon version: 'conmon version 2.1.5, commit: ' cpuUtilization: idlePercent: 86.4 systemPercent: 6.25 userPercent: 7.36 cpus: 6 distribution: distribution: fedora variant: iot version: "37" eventLogger: journald hostname: rockpro64.jwillikers.io idMappings: gidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 uidmap: - container_id: 0 host_id: 1000 size: 1 - container_id: 1 host_id: 100000 size: 65536 kernel: 6.1.6-200.fc37.aarch64 linkmode: dynamic logDriver: journald memFree: 852598784 memTotal: 3994365952 networkBackend: netavark ociRuntime: name: crun package: crun-1.7.2-3.fc37.aarch64 path: /usr/bin/crun version: |- crun version 1.7.2 commit: 0356bf4aff9a133d655dc13b1d9ac9424706cac4 rundir: /run/user/1000/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL os: linux remoteSocket: path: /run/user/1000/podman/podman.sock security: apparmorEnabled: false capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT rootless: true seccompEnabled: true seccompProfilePath: /usr/share/containers/seccomp.json selinuxEnabled: true serviceIsRemote: false slirp4netns: executable: /usr/bin/slirp4netns package: slirp4netns-1.2.0-8.fc37.aarch64 version: |- slirp4netns version 1.2.0 commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383 libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.3 swapFree: 2858676224 swapTotal: 3994021888 uptime: 53h 18m 24.00s (Approximately 2.21 days) plugins: authorization: null log: - k8s-file - none - passthrough - journald network: - bridge - macvlan volume: - local registries: search: - registry.fedoraproject.org - registry.access.redhat.com - docker.io - quay.io store: configFile: /var/home/jordan/.config/containers/storage.conf containerStore: number: 12 paused: 0 running: 11 stopped: 1 graphDriverName: overlay graphOptions: {} graphRoot: /home/jordan/.local/share/containers/storage graphRootAllocated: 123364966400 graphRootUsed: 80927322112 graphStatus: Backing Filesystem: btrfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" imageCopyTmpDir: /var/tmp imageStore: number: 15 runRoot: /run/user/1000/containers volumePath: /home/jordan/.local/share/containers/storage/volumes version: APIVersion: 4.3.1 Built: 1668178831 BuiltTime: Fri Nov 11 09:00:31 2022 GitCommit: "" GoVersion: go1.19.2 Os: linux OsArch: linux/arm64 Version: 4.3.1Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
No
Additional environment details
RockPro64 / aarch64
Fedora IoT 35 - 37
Additional information
Backing storage for containers is running off of an NFS mounted volume and an S3 mounted volume, both of which are mounted via Linux directly.
Several containers and on pod are running on the system, managed via systemd, without problems.
I think this may be related to #10269.
The text was updated successfully, but these errors were encountered: