kind with podman remote

[haarchri]

What happened:
we running podman as sidecar container in gitlab-ci pipelines - we configured the following:

$ export DOCKER_HOST=tcp://podman:8888
$ export DOCKER_DRIVER=fuse-overlayfs
$ export DOCKER_TLS_CERTDIR=""
$ export CONTAINER_CONNECTION=tcp://podman:8888
$ export CONTAINER_HOST=tcp://podman:8888

$ kind create cluster --config test/kind-config.yaml
enabling experimental podman provider
Creating cluster "kind" ...
 • Ensuring node image (kindest/node:v1.21.10) 🖼  ...
 ✓ Ensuring node image (kindest/node:v1.21.10) 🖼
 • Preparing nodes 📦   ...
 ✗ Preparing nodes 📦 
ERROR: failed to create cluster: command "podman run --name kind-control-plane --hostname kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --tmpfs /tmp --tmpfs /run --volume 011549994c011b0585520606d44fc241d77c97d4d4b88e3379e1b68aefec41e3:/var:suid,exec,dev --volume /lib/modules:/lib/modules:ro -e KIND_EXPERIMENTAL_CONTAINERD_SNAPSHOTTER --detach --tty --net kind --label io.x-k8s.kind.cluster=kind -e container=podman --publish=127.0.0.1:45791:6443/tcp -e KUBECONFIG=/etc/kubernetes/admin.conf docker.io/kindest/node:v1.21.10" failed with error: exit status [126](https://gitlab.dev.xxx.sh/aws-ops/xxx/-/jobs/8774672#L126)
Command Output: Error: crun: set xattr for `runc.sha256`: Permission denied: OCI permission denied

podman configuration:

#!/bin/bash
echo "starting podman ..."
unset CONTAINER_HOST
podman system service --time 0 unix:///var/run/docker.sock & 
podman system service --time 0 tcp://0.0.0.0:8888

and containers.conf

[containers]
netns="host"
userns="host"
ipcns="host"
utsns="host"
cgroupns="host"
cgroups="disabled"
default_sysctls = []
log_driver = "k8s-file"
[engine]
cgroup_manager = "cgroupfs"
events_logger="file"
runtime="crun"

any idea what is the problem ?

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• kind version: (use kind version):
• Runtime info: (use docker info or podman info):
• OS (e.g. from /etc/os-release):
• Kubernetes version: (use kubectl version):
• Any proxies or other special environment settings?:

[aojea]

looks more like a podman configuration issue , there are some entries related like containers/podman#14284

[haarchri]

But all other Tools running fine with this podman setup - only kind is with trouble

[BenTheElder]

It's not KIND's job to make podman run work. This podman run command is used on other hosts.

Command Output: Error: crun: set xattr for runc.sha256: Permission denied: OCI permission denied

This is an issue on the podman side. For some unknown reason it's failing to set attributes on one of the image files by the looks of it.