Podman with Kubernetes executor in Gitlab CI

[tvvignesh]

Hi. I am trying to run Podman in Gitlab CI using the Kubernetes executor (GKE with containerd specifically) (https://docs.gitlab.com/runner/executors/kubernetes.html) and am struggling to build the image. Details below.

My Sample Dockerfile:

FROM node:10-alpine

RUN node -v

My Sample CI:

image: "quay.io/podman/stable"

buildah:
  tags:
    - tag1
    - tag2
  variables:
    STORAGE_DRIVER: "vfs"
    IMAGE_TAG: $CI_REGISTRY_IMAGE:edge
  script:
    - whoami
    - podman version
    - echo "Logging into $CI_REGISTRY"
    - podman login -u gitlab-ci-token -p $CI_BUILD_TOKEN $CI_REGISTRY
    - podman build --security-opt seccomp=unconfined --security-opt label=disabled --runtime-flag debug -t ${IMAGE_TAG} .
    - podman images
    - podman push ${IMAGE_TAG}
    - podman logout $CI_REGISTRY

While I wanted to go with the restricted PSP (https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/restricted-psp.yaml) I thought I should get it running first, so I have given all permissions to the PSP exactly using this PSP and have no securityContext in the POD as well: https://raw.githubusercontent.com/kubernetes/website/master/content/en/examples/policy/privileged-psp.yaml

Case 1: All permissions given, and replacing vfs with overlayfs driver

When trying to use overlayfs I did add mount settings like this to gitlab runner but it didn't help:

[[runners.kubernetes.volumes.host_path]]
      name = "buildah"
      mount_path = "/var/lib/containers/"
      read_only = false

Case 2: All permissions given, vfs driver

I am stuck with this error. I did see in this blog: https://blog.42.be/2018/10/buildah-inside-centos-container.html that they are facing the same error and they fixed it by enabling user namespace.

So, I tried adding the --userns flag and set its value to both host and container and it did not work.

I saw a similar issue #3402 using Go and I am facing the same issue with Node running Podman as root.

Also saw somewhat similar issue here with arm (I am using amd64) but not sure if its related: https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4246

Case 3: Trying out security-opt
I did add the security-opt flag as per this comment: containers/buildah#2175 (comment) but that didn't work either and I get the same error as above.

Also tried adding --security-opt seccomp=/usr/share/containers/seccomp.json but that didn't seem to work either.

podman build --security-opt seccomp=unconfined -t ${IMAGE_TAG} .

Case 4: Trying to set the storage driver to fuse
I did try changing storage driver to fuse by setting STORAGE_DRIVER: "fuse" but it didn't work since it said fuse is not supported.

Tried both fuse and /dev/fuse

Is there any way or guide/recommendation to run Podman in Kubernetes executor providing minimum privilege as possible to build images? Currently I am stuck with Case 2 above.

Since I don't have any docker here (everything is running within Containerd since it is using the Kubernetes executor), I don't think that is the problem? I am using the quay.io/podman/stable image as well.

Also referred to https://medium.com/prgcont/using-buildah-in-gitlab-ci-9b529af19e42 but that didn't help due to the errors above.

Any help is appreciated. Thanks.

[mheon]

It looks like whatever you're running Podman inside is blocking the mount syscall (likely via Seccomp or capabilities). I believe you may be able to use Buildah with the VFS graph driver and chroot-only containers to build, but Podman itself cannot function without the ability to mount filesystems. You will have to loosen the permissions on the Kube container that Podman is being run inside if you want to do this.

[tvvignesh]

It looks like whatever you're running Podman inside is blocking the mount syscall (likely via Seccomp or capabilities). I believe you may be able to use Buildah with the VFS graph driver and chroot-only containers to build, but Podman itself cannot function without the ability to mount filesystems. You will have to loosen the permissions on the Kube container that Podman is being run inside if you want to do this.

@mheon Thanks for the quick reply. Has anyone tried a sample of Podman executing in containerd - GKE? I am not using anything else since Gitlab runner just spawns a pod in K8 (https://docs.gitlab.com/runner/executors/kubernetes.html)

Podman is running as root within containerd with all permissions given to it (not sure about filesystem mounting though). Are you talking about something like this - https://docs.gitlab.com/runner/executors/kubernetes.html#using-volumes Is there any volume to be specifically mounted to use inside kubernetes? I did try mounting /var/lib/containers as shown above but that didn't work.

I have no issues using Buildah since I just want to build images, not run containers within my CI pipeline but I thought Podman itself uses buildah underneath. Am I missing something here?

[TomSweeneyRedHat]

@tvvignesh Podman does use Buildah for the build command, but it invokes the buildah bud command with slightly different default options, most notably --layers. @rhatdan any thoughts on what the VFS angle might be doing here?

[tvvignesh]

@TomSweeneyRedHat Hmm. Thanks for the clarification. Not sure how that would affect my build though, but still just tried using buildah rather and I get this:

And this is all I have in my CI: buildah bud -t ${IMAGE_TAG} .

Get the same error as podman with overlayfs

[tvvignesh]

More details on the environment I am running this in:

Server Version: 19.03.2
Storage Driver: overlay2
Backing Filesystem: extfs
Kernel version
5.4.0-1024-gcp
OS image
Ubuntu 18.04.5 LTS
Container runtime version
containerd://1.2.10
kubelet version
v1.18.6-gke.4801
kube-proxy version
v1.18.6-gke.4801

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

The issue is that the default seccomp rules that are being used are blocking the mount syscall and a few others. We do not believe that these should be blocked by default and use less stringent rules for seccomp.json then Docker did. Which allows buildah to be run within a locked down container.

If you switch to using /usr/share/containers/seccomp.json from containers-common, package you should be able to get this to work.

If you switched to CRI-O backend then it should work out of the box.