podman start does not start conmon #16634
Comments
openshift-ci
bot
added
kind/bug
|
there is not any information attached to the issue. Please reopen once you've filled all the requested info and provide a reproducer. |
|
I don't know how to reopen the issue. |
I couldn't reopen the issue, please reopen it as I don't seem to have the access. Otherwise I'll open a new one. |
|
it could be caused by a crash in conmon. Would you be able to verify that? I think something like containers/conmon#361 could be the reason for it |
If 'podman start' would start conmon, regardless if conmon crashes after detaching or not, wouldn't the conmon command with all its options, show up in the 'podman --log-level=debug start' logs, just like in the first 'podman start' command instance in step 2? One other question, quote from conmon README - 'Writes the contents of the container's streams to a log file ', |
I verified, the same behavior with the latest version, conmon is not started/running: |
|
could you run What could happen is that conmon crashed but the container still runs so Also, have you created a new container using the freshly installed conmon so that the first run is using a fixed conmon? |
--max-arg is not a valid option: Also, the OCI 'master' container doesn't have access to /sys/kernel/debug/, it is running on a VM:
crashed when? when I run 'podman start'?
I assume you are referring to podman container that still runs. Is it possible that it runs even if the OCI 'master' container that started it exits?
I killed the podman container after the OCI container restarted but before 'podman start': I also stopped the podman container (on a different test) after the OCI container restarted but before 'podman start':
Yes, the new conmon replaced the old one on the host (a VM), then ran all the testing steps. |
|
from the logs it seems that conmon is started: we need to figure out why it exits. Is there any useful information in the journal? |
Just to clarify, these conmon logs are from the step 2, the first/initial 'podman start' instance. Based on the debug logs, you assume conmon is exiting after 'podman start'.
or
I think I mentioned in one of my questions above that syslog/messages/journal is Not configured on the 'master' OCI container and we need to explicitly setup the --log-path, conmon option, when it is started, I couldn't find a way of doing that from 'podman start' command. Is there any other way? |
|
could the privileged container be messing up with the host somehow? |
|
otherwise I'm not sure how you could end up in that state |
|
Giuseppe, so your guesses don't seem to lead to a solution. Do you mind helping debugging this?
|
it is enough to pass
probably because conmon didn't write the exit status file for the container
could you ping me on IRC so we can debug it together? I am |
Sorry I wasn't clear, I've noticed that the log-level is passed on to conmon, the problem is that conmon doesn't have the log file set as there is no syslog/journald set up on the master/OCI container and I was looking for a way to set the argument for '--log-path' conmon option.
|
|
A friendly reminder that this issue had no activity for 30 days. |
|
@giuseppe Do you have a response? |
|
I don't know what is going on by the details above, it is the first time I hear about such issue so it is likely a local configuration to cause it. I've proposed to help on IRC but got no reply to that. I am closing it since the lack of feedback, please reopen if you've more data or would like me to help debug it as I've offered. |
github-actions
bot
added
the
locked - please file new issue/PR
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
Creating and starting a podman container from an OCI container, using '--root ' option.
Exit and restart the OCI container.
'podman ps' and 'inspect' commands show the podman container running. conmon is not started even after 'podman start'. Podman container PID is dated.
Steps to reproduce the issue:
All podman commands are issued from an OCI container.
Expected.
podman create:
Expected.
podman start, using --log-level=debug option:
Expected.
Restarting the OCI container.
Note that the /persistent directory (used on storage.conf/graphroot and command --root option) is persistent
between the two OCI container runing sessions, as it is bind-mounted on the host path.
/staging directory is local to the OCI container.
container to be in running state is not expected(?)
Getting the old PID is not expected!
comon not starting is not expected!
Using the '--log-level=debug' option again:
'podman start' command succeeds, but looking at the logs, the conmon is not started.
conmon not running is not expected.
Expected conmon to be started by 'podman start' (as on initial 'podman start' command) after OCI is restarted.
No errors on the command debug output either!
Same results (after OCI container restart), on podman ps and inspect commands, inspect returns the wrong container PID:
Not expected.
Retrieving the older container PID is not expected.
The /proc/ does not exist:
Expected conmon to run and with a different container PID after OCI container restart on step 4.
Or at least, if it happens to get the same PID, the '/proc/' should be valid.
Verify that the initial/local file containing the container-PID does not exists:
Describe the results you received:
After restarting the OCI container, 'podman ps' shows the podman container is running and 'podman inspect --format {{.PidFile}}' shows its older PID, from the initial OCI container session before been restarted.
Describe the results you expected:
After restarting the OCI container, 'podman ps' should not show the container in 'running' state.
Not sure what 'podman inspect --format {{.PidFile}}' PID should be returned in the not-running state, but after 'podman start' command, the container PID should match a valid /proc/container-PID entry.
Additional information you deem important (e.g. issue happens only occasionally):
Output of
podman version:(From the OCI container that is used to start podman container(s))
Output of
conmon version:Output of
podman info:storage.conf on OCI container:
Package info (e.g. output of
rpm -q podmanorapt list podmanorbrew info podman):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
No
I can't upgrade the podman version, compatibility issues.
I checked the Troubleshooting Guide.
Additional environment details (AWS, VirtualBox, physical, etc.):
OCI container, which starts the podman container:
The text was updated successfully, but these errors were encountered: