Jenkins is unable to execute podman

[gonrial]

Using this pipeline:

pipeline {
    agent {
        kubernetes {
            cloud 'kubernetes'
            namespace 'jenkins'
            defaultContainer 'jnlp'
            yaml """
kind: Pod
metadata:
  name: jenkins-agent-podman
spec:
  containers:
  - name: jnlp
    image: mgoltzsche/jenkins-jnlp-slave
    imagePullPolicy: Always
    tty: true
  restartPolicy: Never
"""
        }
    }
    stages {
        stage("Podman Check") {
            steps {
                git 'https://github.com/ngallot/docker-python-helloworld'
                sh '''
                pwd
                printenv
                whoami
                cat /etc/passwd
                docker build .
                '''
            }
        }
    }
}

I get the following output

[Pipeline] sh
+ pwd
/home/jenkins/agent/workspace/ins-tests_test_podman
+ printenv
...
JENKINS_HOME=/var/jenkins_home
HOME=/home/jenkins
...
+ whoami
jenkins
+ cat /etc/passwd
...(no podman)
jenkins:x:1000:1000:Jenkins user:/home/jenkins:/bin/sh
+ docker build .
Error: error creating runtime static files directory /podman/.local/share/containers/storage/libpod: mkdir /podman: permission denied
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline

GitHub has been notified of this commit’s build result

ERROR: script returned exit code 125
Finished: FAILURE

I checked and in /etc/containers/storage.conf the graphroot points to /podman/.local/share/containers/storage instead of $HOME/.local/share/containers/storage but even after changing that, podman is still throwing errors:

$ podman --runroot "/home/jenkins/.local/share/containers" --root "/home/jenkins/.local/share/containers" build .
cannot clone: Operation not permitted
Error: cannot re-exec process

I also tested locally using:

$ docker run --rm -it --entrypoint bash --user jenkins mgoltzsche/jenkins-jnlp-slave                              
jenkins@bac07a01b996:~$ git clone https://github.com/ngallot/docker-python-helloworld
Cloning into 'docker-python-helloworld'...
remote: Enumerating objects: 27, done.
remote: Counting objects: 100% (27/27), done.
remote: Compressing objects: 100% (18/18), done.
remote: Total 27 (delta 5), reused 24 (delta 4), pack-reused 0
Unpacking objects: 100% (27/27), done.
jenkins@bac07a01b996:~$ docker --runroot "/home/jenkins/.local/share/containers" --root "/home/jenkins/.local/share/containers" build docker-python-helloworld
cannot clone: Operation not permitted
Error: cannot re-exec process
jenkins@bac07a01b996:~$ 

Any ideas what can be happening?

[longwuyuan]

the binaries copied over fro the other image are musl built. Will they work against gcc ;

_$ sudo docker run --rm -ti --name podman mgoltzsche/podman ldd podman                            
/lib/ld-musl-x86_64.so.1: cannot load podman: No such file or directory

[mgoltzsche]

@gonrial Regarding the wrong home directory: This happens because the configuration changed meanwhile within the podman image and the Dockerfile refers to the latest tag - I didn't touch the jenkins image here for some time. I ll try to make the podman image user/home-agnostic. Alternatively a storage.conf specific to this image could be added here.
The permission error happens because your outer docker container is missing some capabilities. It should work if you add the --privileged option to docker run (the linux kernel requires a process to have CAP_SYS_ADMIN in order to switch namespaces). Granting privileges to the container also means that a process could break out of it but at least in this case it is still just the unprivileged jenkins user - not root.

@longwuyuan the podman binaries are statically linked which is why they also run with other linux distributions - I wonder why it prints "cannot load podman" in your case.... I ll have a look.

[longwuyuan]

thanks. the goal was to run unprivileged hence the choice of podman instead of dind. would you know if podman can be run without the jnlp image being privileged

[mgoltzsche]

@longwuyuan First of all you need to distinguish between running a container as unprivileged (non-root) user (known as "rootless container") and running it with extended privileges/capabilities. While you cannot run docker as unprivileged user you can do so using podman - that's a big advantage already. However even if you run podman as unprivileged user it still requires some additional capabilities (at least CAP_SYS_ADMIN) which are disabled within a docker container by default unless you provide the --privileged flag to it or specify the required capabilities explicitly. (As a regular non-root user on a host system usually you have CAP_SYS_ADMIN by default but that doesn't mean that you have root privileges.) However giving a container such capabilities allows it to break out and access the host (as that unprivileged user).
This limitation of running containers with additional capabilities originates from the linux kernel - not from podman/runc. Therefore currently (and in the near future) you won't find any other truely native linux container solution that is able to run without those additional capabilities - however eventually it works with some features emulated while sacrificing performance and isolation.

So you can run the jenkins agent container as unprivileged user with podman but you need to be aware that it would be possible for a process within the jenkins container to break out. (if you wanted to be really sure that this cannot affect other processes/data on the same host you'd need to run the jenkins container with a uid/gid that is unique on that host.)

Also see e.g. containers/podman#4131

Oh and I realize the "cannot load podman" error originated from ldd because you didn't provide the correct podman binary path - so nothing to worry in that matter.

[longwuyuan]

I read through the issue and know better now.
thank you very much for the info.

I was attempting to use this image https://github.com/longwuyuan/jenkins-jnlp-agent-podman, to build this app https://github.com/longwuyuan/docker-python-helloworld , with jenkins installed using the stable helm chart (and hence shipping the kubernetes plugin out of the box) and the agent launching as a pod due to this pipeline https://github.com/longwuyuan/docker-python-helloworld/blob/master/Jenkinsfile .. Fails with issues you are referring to

[longwuyuan]

I have been advised to use buildah containers/podman#8190

[mgoltzsche]

I am closing this issue because meanwhile I fixed the podman image so that it is home dir agnostic again.

@longwuyuan while buildah is fine if you just want to build a container image it is not sufficient if you want to have a full docker CLI replacement.