Rootless Podman won't work since version 3.1.0 #9936
Comments
openshift-ci-robot
added
the
kind/bug
|
I'm also experiencing this issue, and just thought I'd note that I can't find any pattern when looking at various image manifests. archlinux manifest {
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1801,
"digest": "sha256:3de742be9254c8423b72d08dde70cc049d9897ea8de1a981aa77b9071e41ad53"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 134275179,
"digest": "sha256:10756994dc1943314a638a8a9f7808b269aff96c2890521a68699803a98f792f"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 6660,
"digest": "sha256:5bb50848eab8d3d80a48b3769ef342097f57881b1ef86826e898c43ee4dd2460"
}
]
}ubuntu manifest {
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 3316,
"digest": "sha256:26b77e58432b01665d7e876248c9056fa58bf4a7ab82576a024f5cf3dac146d6"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 28569016,
"digest": "sha256:a70d879fa5984474288d52009479054b8bb2993de2a1859f43b5480600cecb24"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 848,
"digest": "sha256:c4394a92d1f8760cf7d17fee0bcee732c94c5b858dd8d19c7ff02beecf3b4e83"
},
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 187,
"digest": "sha256:10e6159c56c084c858f5de2416454ac0a49ddda47b764e4379c5d5a147c9bf5f"
}
]
}alpine manifest {
"schemaVersion": 2,
"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
"config": {
"mediaType": "application/vnd.docker.container.image.v1+json",
"size": 1472,
"digest": "sha256:49f356fa4513676c5e22e3a8404aad6c7262cc7aaed15341458265320786c58c"
},
"layers": [
{
"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
"size": 2811947,
"digest": "sha256:ca3cd42a7c9525f6ce3d64c1a70982613a8235f0cc057ec9244052921853ef15"
}
]
} |
|
I got the same issue, same behaviours as OP. |
|
@rhatdan PTAL |
|
i am getting this error Getting image source signatures |
|
There seems to be an at least related issue with consistent environment variable expansion in When running additional commands like e.g. Used package, distro, versions and relevant configuration parts are identical to those of the OP. |
|
@giuseppe Ideas? Are you seeing issues with SELinux or seccomp? |
|
Could you make sure you have the latest fuse-overlayfs installed. |
|
@giuseppe could this be these platforms attempting to use native overlay? |
|
@rhatdan I don't use SELinux. For me, looks like a regression, since podman |
|
I tried this $HOME variables expanded correctly, but indeed podman CLI is hanging even on info command |
|
Find, what I don't know is if these are all the same issues. Is everyone running on arch, has everyone modified the storage.conf? |
Regarding the
I am running Arch. I have modified I agree that this seems like two separate problems:
|
@rhatdan OP here. Yes, using fuse-overlayfs 1.5 0. |
|
@rhatdan No config files modified from vanilla.
|
|
What does podman info show for storage? |
|
|
I think this commit ec1651f broke something. |
unprivileged users cannot use the trusted.* xattrs. Since for rootless we always mount overlay with userxattr, we can just check if running in rootless mode and use user.* instead of trusted.*. Closes: containers/podman#9936 Signed-off-by: Giuseppe Scrivano <[email protected]>
the error I am seeing "Error processing tar file(exit status 1): operation not permitted" happens at pull time. It turns to be another issue in containers/storage with native rootless overlay: containers/storage#872 |
unprivileged users cannot use the trusted.* xattrs. Since for rootless we always mount overlay with userxattr, we can just check if running in rootless mode and use user.* instead of trusted.*. Closes: containers/podman#9936 Signed-off-by: Giuseppe Scrivano <[email protected]>
|
another issue is fixed with: containers/storage#867 |
unprivileged users cannot use the trusted.* xattrs. Since for rootless we always mount overlay with userxattr, we can just check if running in rootless mode and use user.* instead of trusted.*. Closes: containers/podman#9936 Signed-off-by: Giuseppe Scrivano <[email protected]>
unprivileged users cannot use the trusted.* xattrs. Since for rootless we always mount overlay with userxattr, we can just check if running in rootless mode and use user.* instead of trusted.*. Closes: containers/podman#9936 Signed-off-by: Giuseppe Scrivano <[email protected]>
unprivileged users cannot use the trusted.* xattrs. Since for rootless we always mount overlay with userxattr, we can just check if running in rootless mode and use user.* instead of trusted.*. Closes: containers/podman#9936 Signed-off-by: Giuseppe Scrivano <[email protected]>
unprivileged users cannot use the trusted.* xattrs. Since for rootless we always mount overlay with userxattr, we can just check if running in rootless mode and use user.* instead of trusted.*. Closes: containers/podman#9936 Signed-off-by: Giuseppe Scrivano <[email protected]>
unprivileged users cannot use the trusted.* xattrs. Since for rootless we always mount overlay with userxattr, we can just check if running in rootless mode and use user.* instead of trusted.*. Closes: containers/podman#9936 Signed-off-by: Giuseppe Scrivano <[email protected]>
|
Yo, |
|
containers/storage#872 should be ready to merge, but containers/storage#867 does not look like it's ready yet. Once both are, we can cut a fresh release of c/storage and get it vendored into Podman for a 3.1.1. In the meantime, anyone who can manually apply the two and do a scratch-build to verify this actually resolves the issue would be greatly appreciated - I can't reproduce this myself so I can't verify the fix. |
|
I can confirm containers/storage#872 fixes λ libpod master» ./bin/podman info
host:
arch: amd64
buildahVersion: 1.20.0
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: /usr/bin/conmon is owned by conmon 1:2.0.27-1
path: /usr/bin/conmon
version: 'conmon version 2.0.27, commit: 65fad4bfcb250df0435ea668017e643e7f462155'
cpus: 8
distribution:
distribution: arch
version: unknown
eventLogger: journald
hostname: anathema
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 5.11.11-arch1-1
linkmode: dynamic
memFree: 1332187136
memTotal: 16536141824
ociRuntime:
name: runc
package: /usr/bin/runc is owned by runc 1.0.0rc93-2
path: /usr/bin/runc
version: |-
runc version 1.0.0-rc93
commit: 12644e614e25b05da6fd08a38ffa0cfe1903fdec
spec: 1.0.2-dev
go: go1.16.2
libseccomp: 2.5.1
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
selinuxEnabled: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: /usr/bin/slirp4netns is owned by slirp4netns 1.1.9-1
version: |-
slirp4netns version 1.1.9
commit: 4e37ea557562e0d7a64dc636eff156f64927335e
libslirp: 4.4.0
SLIRP_CONFIG_VERSION_MAX: 3
libseccomp: 2.5.1
swapFree: 0
swapTotal: 0
uptime: 26h 27m 38.96s (Approximately 1.08 days)
registries: {}
store:
configFile: /home/fox/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/fox/.local/share/containers/storage
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 0
runRoot: /run/user/1000/containers
volumePath: /home/fox/.local/share/containers/storage/volumes
version:
APIVersion: 3.2.0-dev
Built: 1617832824
BuiltTime: Thu Apr 8 00:00:24 2021
GitCommit: 0e67053b9a26f20e5ccbffdcc5e7a84254ca16b8-dirty
GoVersion: go1.16.3
OsArch: linux/amd64
Version: 3.2.0-dev
Tiny bit manual patching to build. λ libpod master» git diff Makefile go.mod
diff --git a/Makefile b/Makefile
index a70e07991..a0fd2611d 100644
--- a/Makefile
+++ b/Makefile
@@ -41,7 +41,7 @@ PRE_COMMIT = $(shell command -v bin/venv/bin/pre-commit ~/.local/bin/pre-commit
# our target (bin/podman{,-remote}), a rebuild is triggered.
SOURCES = $(shell find . -path './.*' -prune -o \( -name '*.go' -a ! -name '*_test.go' \) -print)
-BUILDFLAGS := -mod=vendor $(BUILDFLAGS)
+BUILDFLAGS := $(BUILDFLAGS)
BUILDTAGS_CROSS ?= containers_image_openpgp exclude_graphdriver_btrfs exclude_graphdriver_devicemapper exclude_graphdriver_overlay
CONTAINER_RUNTIME := $(shell command -v podman 2> /dev/null || echo docker)
diff --git a/go.mod b/go.mod
index 52d632b46..699cb99dd 100644
--- a/go.mod
+++ b/go.mod
@@ -71,3 +71,5 @@ require (
k8s.io/api v0.20.5
k8s.io/apimachinery v0.20.5
)
+
+replace github.com/containers/storage => ../storage
|
|
When we can expect podman 3.1.1 release into alpine community ? |
|
I would expect sometime next week. We need a fresh c/storage release cut and vendored into Podman, then we can cut a fresh release. |
github-actions
bot
added
the
locked - please file new issue/PR
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Since v3.1.0, rootless Podman is not working with the same configuration that worked in v3.0.1 for some images.
It is hanging with zero output (only when ran in rootless mode!) with
rootless_storage_pathdefined with "$HOME" variable. Ifrootless_storage_pathis commented out or set to an user's home path (instead of $HOME variable), it sort of works. It is as if $HOME was being used literally, instead of as a variable.When I say "sort of works" it is because it still gives "operation not permitted" with some Docker images, like archlinux or ubuntu. However, I tested with alpine and fedora, and it works. See error outputs below.
Steps to reproduce the issue:
enable
rootless_storage_pathsetting in/etc/containers/storage.confwith the following valuerun
podman pull docker.io/archlinuxas normal userthe CLI should be hanging, so run Ctrl+C to cancel the previous command
now set
rootless_storage_pathwith the home directory, say "/home/foo/"run
podman pull docker.io/archlinuxas normal user againlayers are download, but it ends up with "operation not permitted" error message
Describe the results you received:
Terminal is hanging when
rootless_storage_pathis set and has "$HOME" in its path:If "$HOME" is replaced with real valid path like "/home/foo", I get the following output:
But not all images:
Describe the results you expected:
Be able to use Podman in rootless mode for all images, mainly Arch Linux
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:Output of
podman info --debug:Package info (e.g. output of
rpm -q podmanorapt list podman):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?
Yes (the commit tested was 72eb000) and yes
Additional environment details (AWS, VirtualBox, physical, etc.):
Its a physical machine, running Arch Linux 64-bit, with containers-common installed.
Here is my rootless mode settings (See Podman in ArchWiki for the guide followed):
kernel.unprivileged_userns_clone=1already set by the stock Linux kernel in Arch Linuxrafael:100000:65536(my username and group)The text was updated successfully, but these errors were encountered: