-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Podman 2.2.1 rootless: cannot play podman-generated YAML file #8710
Comments
Can you provide more details - are you running Podman as root, or rootless? Does the container you are running |
Hi @mheon , thanks for the advice. I am testing it with crun now. Yes, in the second scenario I was able to start the container with I need some time for crun. I'll let you know the results soon. |
Hi @mheon. Scenario 1:
Here I end up having a pod in "Degraded" status where only the infra container is running, the main container is "Created".
Scenario 2: I cannot find more information in the logs, other than |
Hi, I did some more research - this time only with the original YAML file (scenario 1).
it leads to this error message:
The same with
|
Can you check the journal for anything from |
All I can see in the journal is what I see when I |
I think I have a similar problem. Still new to podman and was developing a pod containing mssql-2019, 2x nginx, aspnetcore-3.. Had a problem with volumes and tried podman 2.2 (update from 2.1), now nothing works any more. Well, thought "let's downgrade back to 2.1". But:
So, as I am stuck anyway.... how can I help? All containers except the asp.net core behave very strange. This is the log of mssql2019 when started via
note the contradiction between the statements about the user running the container process. When starting the container directly via
both nginx's aren't outputting anything at all before dying. However, those can be started directly via Fedora 33 btw. |
Can you try to compile it with the latest code branch? |
in general same issue there on fresh Fedora 33 and stock Podman 2.2.1 install. Rootless. WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument and [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied I tried to
But apiVersion: v1
kind: Pod
metadata:
name: web
spec:
containers:
- name: nginx
image: docker.io/nginx:latest fails to run. UPDATE: 1 apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
hostPort: 8080 Strange... UPDATE: 2 resources:
requests:
cpu: 1
memory: 1Gi
limits:
cpu: 2
memory: 1Gi
Error: error starting some containers: some containers failed UPDATE: 3 UPDATE: 4 $ podman exec -it web-nginx bash
root@web:/# ls /var/www/html\:Z/
ls: cannot open directory '/var/www/html:Z/': Permission denied |
Both of the debug logs you listed are warnings and are not fatal. Neither
is the cause of your issues. Please provide the full logs.
The restarting looks like a container with restart policy set restarting
automatically after failure.
…On Thu, Dec 24, 2020 at 09:21 Dzintars Klavins ***@***.***> wrote:
How can this behaviour be explained? Just spam the podman ps -a and see
the random status of the nginx container.
[image: image]
<https://user-images.githubusercontent.com/547420/103093786-086b5f00-4604-11eb-833c-c0752828f9b3.png>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#8710 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB3AOCCKEKQP5TXVV2CUZMTSWNFABANCNFSM4U2ONDFQ>
.
|
Strong suspicion from what you described: Podman is working, but the app in
the container is failing and exiting immediately after it starts
…On Thu, Dec 24, 2020 at 09:27 Dzintars Klavins ***@***.***> wrote:
journalctl -xe -f just goes crazy. Flooded with
░░ The unit UNIT has successfully entered the 'dead' state.
Dec 24 16:25:40 workstation podman[1106230]: 2020-12-24 16:25:40.48915476 +0200 EET m=+0.225365973 container start b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx, maintainer=NGINX Docker Maintainers ***@***.***>)
Dec 24 16:25:40 workstation podman[1106255]: 2020-12-24 16:25:40.544824949 +0200 EET m=+0.042711708 container died b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx)
Dec 24 16:25:40 workstation vault[1479]: 2020-12-24T16:25:40.571+0200 [INFO] http: TLS handshake error from 192.168.1.2:41868: write tcp 192.168.1.2:8200->192.168.1.2:41868: write: connection reset by peer
Dec 24 16:25:40 workstation podman[1106255]: 2020-12-24 16:25:40.587223525 +0200 EET m=+0.085110298 container restart b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx, maintainer=NGINX Docker Maintainers ***@***.***>)
Dec 24 16:25:40 workstation systemd[2133]: Started libcrun container.
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 206900.
Dec 24 16:25:40 workstation podman[1106255]: 2020-12-24 16:25:40.717075682 +0200 EET m=+0.214962468 container init b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx, maintainer=NGINX Docker Maintainers ***@***.***>)
Dec 24 16:25:40 workstation audit[1106280]: AVC avc: denied { read } for pid=1106280 comm="nginx" name="nginx.conf" dev="md126p3" ino=24359 scontext=system_u:system_r:container_t:s0:c492,c1012 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0
Dec 24 16:25:40 workstation systemd[2133]: libpod-b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf.scope: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit UNIT has successfully entered the 'dead' state.
Dec 24 16:25:40 workstation podman[1106255]: 2020-12-24 16:25:40.736424453 +0200 EET m=+0.234311233 container start b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx, maintainer=NGINX Docker Maintainers ***@***.***>)
Dec 24 16:25:40 workstation podman[1106282]: 2020-12-24 16:25:40.815438293 +0200 EET m=+0.042481318 container died b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx)
Dec 24 16:25:40 workstation podman[1106282]: 2020-12-24 16:25:40.850864503 +0200 EET m=+0.077907529 container restart b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx, maintainer=NGINX Docker Maintainers ***@***.***>)
Dec 24 16:25:40 workstation systemd[2133]: Started libcrun container.
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 206905.
Dec 24 16:25:40 workstation podman[1106282]: 2020-12-24 16:25:40.968409017 +0200 EET m=+0.195452039 container init b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx, maintainer=NGINX Docker Maintainers ***@***.***>)
Dec 24 16:25:40 workstation audit[1106306]: AVC avc: denied { read } for pid=1106306 comm="nginx" name="nginx.conf" dev="md126p3" ino=24359 scontext=system_u:system_r:container_t:s0:c492,c1012 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0
Dec 24 16:25:40 workstation systemd[2133]: libpod-b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf.scope: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit UNIT has successfully entered the 'dead' state.
Dec 24 16:25:41 workstation podman[1106282]: 2020-12-24 16:25:41.009143843 +0200 EET m=+0.236186895 container start b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx, maintainer=NGINX Docker Maintainers ***@***.***>)
Dec 24 16:25:41 workstation podman[1106308]: 2020-12-24 16:25:41.072882434 +0200 EET m=+0.042637370 container died b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx)
Dec 24 16:25:41 workstation podman[1106308]: 2020-12-24 16:25:41.105164991 +0200 EET m=+0.074919917 container restart b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx, maintainer=NGINX Docker Maintainers ***@***.***>)
Dec 24 16:25:41 workstation systemd[2133]: Started libcrun container.
░░ Subject: A start job for unit UNIT has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit UNIT has finished successfully.
░░
░░ The job identifier is 206910.
Dec 24 16:25:41 workstation podman[1106308]: 2020-12-24 16:25:41.214473002 +0200 EET m=+0.184227933 container init b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf (image=docker.io/nginx:1.14.2, name=web-nginx, maintainer=NGINX Docker Maintainers ***@***.***>)
Dec 24 16:25:41 workstation audit[1106331]: AVC avc: denied { read } for pid=1106331 comm="nginx" name="nginx.conf" dev="md126p3" ino=24359 scontext=system_u:system_r:container_t:s0:c492,c1012 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0
Dec 24 16:25:41 workstation systemd[2133]: libpod-b34add2bcac77c5617692071c2c8868a92ba514cc256290a56a88dcacf1f8cdf.scope: Succeeded.
░░ Subject: Unit succeeded
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#8710 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB3AOCGXK62XJSXQ5EOXRNLSWNFWPANCNFSM4U2ONDFQ>
.
|
There are no errors out of Podman here - so it's definitely the app in the container. |
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied
....
volumeMounts:
- name: public
mountPath: /var/www/html
readOnly: true What is this |
If you are running stock podman from Fedora 33, you have to specify the full nginx command due to the two bugs I already mentioned.
|
@oleastre Yes. Seems you are right. I tried this on Jenkins container and got it finally running.
» podman exec -it jenkins-server bash
jenkins@jenkins:/$ cat /etc/passwd | grep jenkins
jenkins:x:1000:1000::/var/jenkins_home:/bin/bash
exit or read the docs or examine
» podman unshare chown 1000:1000 -R /home/dzintars/containers/jenkins/volume/jenkins_home
» podman inspect jenkins-server
command:
- /sbin/tini
- --
- /usr/local/bin/jenkins.sh
- /sbin/tini
- --
- /usr/local/bin/jenkins.sh
volumeMounts:
- name: jenkins-home
mountPath: /var/jenkins_home:Z
securityContext:
runAsUser: 1000
allowPrivilegeEscalation: true
capabilities: {}
privileged: false
readOnlyRootFilesystem: false
seLinuxOptions: {} I hope i didn't miss anything there as i did also other system tweaks. Will check these steps on other containers tomorrow. |
OK, I managed to take a step ahead.
Here's how my yaml file looks like:
|
As the message says, the port (you're obscured the exact number) you want to bind to seems to be in use on the host. Likely another program on your system is already using it? |
hi @mheon , there was no other process, as I only use that port for podman. If it had been podman using the port I would have got a "pod is already there"-kind of message.
As it complains for some OOM thing can it not be related to this: #7853 ? I understand it's a new feature in 2.2.0 -- 2.1.1 still works for me. |
Both of those are expected warnings in rootless mode and are not really concerning. |
So it fails for some other reason, doesn't it?
So the |
Can you provide the full log? I'm not seeing the actual case of that error anywhere. |
perhaps you can provide a generic yaml file that reproduces for you so we can run it |
OK, so now I tried to run nginx - I removed the volume part from the file.
current versions:
|
Found (part) of the problem: |
After patching Podman to actually report errors, I'm getting the following: |
I don't think I've ever seen this one before... |
Error handling fix in #8917 |
(This will not resolve your issue, but it will print the actual errors that are occurring) |
Think the problem is being caused by an invalid selinux label being generated, when i use play kube with a similar setup to above the container ends up with a label something like
which for some reason contains 2 types and role is missing. so i think there may be an issue with the mapping from the selinux config in the kube yaml and the context object on the container. |
had a look through some code last night and would seem to be a problem in pkg/specgen/generate/kube/kube.go, regardless of what's put in the selinux options in the yaml file it always ends up in the role position in the label. will try to submit a PR this week for this.
|
SGTM! Thanks for taking this one. |
Fix has merged |
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
YAML files generated by podman cannot be run using
podman play kube
.Steps to reproduce the issue:
When I play it I get:
It creats the pod but the main container is not running.
When I try to
podman start myapp
I get this error:Then I exported it :
podman generate kube <pod_id> > generate.yaml
. Now the YAML file looks similar to this:But when I try to play it although the container starts the process inside has no access to the volume (permission denied).
Describe the results you expected:
I understand the problem is somewhere around the SELinux settings (SELinux is in enforcing mode).
I would expect the spc_t type to continue working or at least the generated YAML file should be playable without modification.
Output of
podman version
:Output of
podman info --debug
:Package info (e.g. output of
rpm -q podman
orapt list podman
):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?
Yes
The text was updated successfully, but these errors were encountered: