podman top -eo failed with error

[LexNetAT]

/kind bug

Description

"podman top ctrID -eo user,pid,comm" output a error message in place of the ps output
podman should support the parameter -eo as well as docker.

Steps to reproduce the issue:

Use a container id from the podman ps command, in this example the id 9526245a1f34.
podman top 9526245a1f34 -eo user,pid,comm

Describe the results you received:

Error: error executing ps(1) in the container: executable file not found in $PATH: No such file or directory: OCI runtime command not found error

Describe the results you expected:

PID COMMAND
12201 cat

With the docker command you get

docker top 9526245a1f34 -eo pid,comm
PID COMMAND
12201 cat

Additional information you deem important (e.g. issue happens only occasionally):
Output of podman version:

Version: 2.1.1
API Version 2.0.0
Go Version: go1.13.15
Buildt: Wed Sep 30 21:51:15 2020
OS/Arch: linux/amd64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

[mheon]

@vrothberg PTAL

@LexNetAT Can you provide more details on the container this is being run in - how it was created, what image it's using, etc.

[vrothberg]

podman top is, in some way, a hybrid. While it supports all the advanced descriptors from containers/psgo it also needs to be compatible with Docker which is executing ps(1). In this case, the -eo will cause Podman to fallback to executing ps -eo ... in the container and it looks like ps isn't present (or in the path).

I suggest to call podman top ... pid,comm instead.

[LexNetAT]

A Debian and a Red Hat ubi8 Images were used for testing

[user@nb ~]$ podman pull debian:buster-20201012
Trying to pull docker.io/library/debian:buster-20201012...
Getting image source signatures
Copying blob e4c3d3e4f7b0 skipped: already exists
Copying config 1510e85017 done
Writing manifest to image destination
Storing signatures
1510e850178318cd2b654439b56266e7b6cbff36f95f343f662c708cd51d0610

[user@nb ~]$ podman images
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.io/library/debian latest 1510e8501783 2 days ago 119 MB
docker.io/library/debian buster-20201012 1510e8501783 2 days ago 119 MB

[user@nb ~]$ podman run -t -d docker.io/library/debian:buster-20201012 cat
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
35cd37a5304d5ad582b1c7dca1c16d1ba9eed6874a002149fe41681a6281b09b

[user@nb ~]$ podman ps
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
35cd37a5304d docker.io/library/debian:buster-20201012 cat 11 seconds ago Up 11 seconds ago romantic_bardeen

[user@nb ~]$ podman top 35cd37a5304d -eo pid,comm
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: error executing ps(1) in the container: executable file ps not found in $PATH: No such file or directory: OCI runtime command not found error

[user@nb ~]$ podman top 35cd37a5304d pid,comm
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
PID COMMAND
1 cat

The same Image with Docker
[root@debian ~] docker version
Client:
Version: 18.09.1
API version: 1.39
Go version: go1.11.6
Git commit: 4c52b90
Built: Sun, 14 Jun 2020 22:12:29 +0200
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.09.1
API version: 1.39 (minimum version 1.12)
Go version: go1.11.6
Git commit: 4c52b90
Built: Sun Jun 14 20:12:29 2020
OS/Arch: linux/amd64
Experimental: false

[root@debian ~] docker run -t -d debian:buster-20201012 cat
56931c1702bca0a53c7d4935e0229c5c7c45235f5cab43457bafb712300eeb7e
[root@debian ~] docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
56931c1702bc debian:buster-20201012 "cat" 5 seconds ago Up 4 seconds dreamy_jackson
[root@debian ~] docker top 56931c1702bc -eo pid,comm
PID COMMAND
3946 cat
[root@debian ~] docker top 56931c1702bc pid,comm
Error response from daemon: ps: error: process ID list syntax error

When the docker image contains the ps command, we get

[user@nb ~]$ podman ps
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
677708153f9b localhost/debian_ps:bullseye cat 3 seconds ago Up 3 seconds ago nervous_zhukovsky

[user@nb ~]$ podman top 677708153f9b -eo pid,comm
PID COMMAND
1 cat
7 ps
[user@nb ~]$ podman top 677708153f9b pid,comm
PID COMMAND
1 cat

With the -eo paramter, the ps command is also displayed in the process list.
Without the -eo parameter in podman, the process list is the same as in docker with the -eo parameter.

The docker command works only with parameter -eo. Without the paramter -eo I get an error message.

The parameters of the docker and podman commands should have the same behavior.

We use the docker images in a Jenkins Pipeline. Jenkins use the docker top command, when running a docker image.

[vrothberg]

Thanks for providing the details, @LexNetAT.

It seems like Docker is executing ps(1) on the host (and I'm fairly certain that was not the case initially). This implies that the PIDs relate to the host's namespace. If you want to display this information with Podman, you can use the host descriptors (see podman-top man page). For instance, hpid for the host PID or huser for the host user.

I think we can fallback to executing ps(1) on the host instead of in the container but that is unreliable to parse.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[zhangguanzhang]

podman top relies on the ps command in the container

[rhatdan]

I don't believe we should fix this, but rely on ps inside of the container. I don't think it is worth the effort to attempt to use ps on the host.

[vrothberg]

Reopening as I think the docs (help message, man pages, etc.) should reflect the difference to Docker.

If users want to find out host information of container processes, podman top + the new descriptors is the right thing.

[vrothberg]

#8449

[bverkron]

We use the docker images in a Jenkins Pipeline. Jenkins use the docker top command, when running a docker image.

Did you ever find a solution to this @LexNetAT? We're running into the same problem (trying to swap out docker for podman) and this is a current hang up for us.

Not sure how to get around it without forking the Docker Pipeline (aka "workflow") plugin to remove the -eo, which isn't a great solution.

https://github.com/jenkinsci/docker-workflow-plugin/blob/c7dedefcf68f6ce1320c8aaaeb831628ea21a086/src/main/java/org/jenkinsci/plugins/docker/workflow/client/DockerClient.java#L150

[rhatdan]

This should have been fixed two years ago? What problem are you seeing? Are you using an older version of Podman?

[bverkron]

This should have been fixed two years ago?

Interesting, based on the comments above is sounded like it would not be fixed, but I guess it was via #8449?

What problem are you seeing?

Same error as @LexNetAT described. When Jenkins tries to run docker top 9526245a1f34 -eo pid,comm it throws
Error: error executing ps(1) in the container: executable file not found in $PATH: No such file or directory: OCI runtime command not found error

Are you using an older version of Podman?

Podman is out of date but according to https://podman.io/releases/ still from earlier this year, not multiple years old.

$ podman version
Client:       Podman Engine
Version:      4.0.2
API Version:  4.0.2
Go Version:   go1.17.7

Built:      Tue Apr 19 03:16:32 2022
OS/Arch:    linux/amd64

I can attempt to update it next week but sounds like it should have the fix already.

[rhatdan]

Yes, if you still have a problem open a new issue with updated data. We don't like to go back to old issues, since it is possible/likely you are seeing a different issue.

[bverkron]

I was able to upgrade podman but only to version 4.1.1. I'm on RHEL 8.6 and that seems to be the newest available.

$ podman version
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.17.7
Built:        Mon Jul 11 07:56:53 2022
OS/Arch:      linux/amd64

The upgrade did not help but I did discover that I can bet past the ps error by running the top command with sudo but get hung up by another error that I'm not sure what to do with.

$ docker top 391aa9539b10d37708896b0de0b2760014e0d28547f0087b8c49e283679653a6 -eo pid,comm
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: error executing ps(1) in the container: runc: exec failed: container_linux.go:380: starting container process caused: exec: "ps": executable file not found in $PATH: OCI runtime attempted to invoke a command that was not found

$ sudo docker top 391aa9539b10d37708896b0de0b2760014e0d28547f0087b8c49e283679653a6 -eo pid,comm
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
Error: unable to lookup requested container: no container with name or ID "391aa9539b10d37708896b0de0b2760014e0d28547f0087b8c49e283679653a6" found: no such container

The container definitely exists, is running, and I can attach to it and run commands inside of it. Not sure what to make of that error. I assume that getting past the initial ps error by using sudo means something isn't setup correctly on my side with permissions. I can attempt to troubleshooting this elsewhere online. Just wanted to toss something in here as a follow up in case others run across this thread like I did.

[vrothberg]

Please open a new issue with a reproducer. As Dan mentions above, recycling old issues does not work well.