podman exec fails #2968

panzerdev · 2019-04-18T14:04:15Z

BUG REPORT
/kind bug

Can't exec command in running container

After starting a container I can't use exec to execute a command int the container.

Steps to reproduce the issue:

podman run --name ubuntu -it registry.hub.docker.com/library/ubuntu bash
Open new terminal window/tab
podman exec -it ubuntu bash

Describe the results you received:

mpa@mpa-MacBookPro:~$ podman exec -it ubuntu bash
exec failed: container_linux.go:345: starting container process caused "process_linux.go:91: executing setns process caused \"exit status 22\""
Error: exit status 1

Describe the results you expected:
A bash promt inside the container

Additional information you deem important (e.g. issue happens only occasionally):
Happens always

Output of podman version:

Version:            1.3.0-dev
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

Output of podman info --debug:

mpa@mpa-MacBookPro:~$ podman info --debug
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xf8 pc=0x111dda3]

goroutine 1 [running]:
github.com/containers/libpod/libpod.(*Runtime).storeInfo(0xc4208e4b60, 0x2417fb0, 0x0, 0x0)
	/build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/libpod/info.go:119 +0x53
github.com/containers/libpod/libpod.(*Runtime).Info(0xc4208e4b60, 0x16aeea8, 0xc4208e4b60, 0x0, 0x0, 0x1661be6)
	/build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/libpod/runtime.go:1075 +0x110
main.infoCmd(0x23f5ac0, 0x0, 0x0)
	/build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/cmd/podman/info.go:59 +0xc7
main.glob..func49(0x23934c0, 0xc420374d40, 0x0, 0x1, 0x0, 0x0)
	/build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/cmd/podman/info.go:32 +0x87
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x23934c0, 0xc4200cc080, 0x1, 0x1, 0x23934c0, 0xc4200cc080)
	/build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x2394c80, 0x6, 0x24199e0, 0xc4201d7100)
	/build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(0x2394c80, 0xc420704370, 0xc4200bc058)
	/build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
	/build/podman-OeFyYA/podman-1.3.0/src/github.com/containers/libpod/cmd/podman/main.go:150 +0x3e

Additional environment details (AWS, VirtualBox, physical, etc.):
Ubuntu 16.04 installen on a MacBook Pro 15 without any virutalization.

The text was updated successfully, but these errors were encountered:

rhatdan · 2019-04-18T14:06:59Z

Where did you get the software for this? Do you have an up 2 date version of runc.
@lsm5 PTAL

panzerdev · 2019-04-18T14:10:01Z

I followed the Ubuntu instructions here
https://github.com/containers/libpod/blob/master/install.md
and had the latest docker installed

mpa@mpa-MacBookPro:/tmp$ runc -v
runc version 1.0.0-rc6+dev
commit: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
spec: 1.0.1-dev

mheon · 2019-04-18T14:15:39Z

Info seems to be segfaulting on trying to dereference r.store. Given the way the PPA works, this is probably master, which means it has the single rootless userns patch, which I think means we always configure the rootless userns.

mheon · 2019-04-18T14:28:02Z

@panzerdev Can you try those commands as root, and see if they work?

panzerdev · 2019-04-18T14:38:02Z

If I run podman as root for the run and exec command it works a expected.

Have I overlooked the instruction to run it as root or so?

mheon · 2019-04-18T14:40:29Z

No, this sounds like a bug, but one specific to our rootless code - we need to do more work to get Podman running without root (setting up a user namespace so we can gain limited versions of certain capabilities), and some piece of that seems to be breaking.

baude · 2019-05-29T15:35:59Z

@panzerdev is this working for you now? @giuseppe ptal if not

panzerdev · 2019-05-30T21:08:20Z

@baude I am running

Version:            1.3.2-dev
RemoteAPI Version:  1
Go Version:         go1.10.4
OS/Arch:            linux/amd64

and it is sadly still the same issue. Do I need to change something about my installation?

mheon · 2019-05-31T00:01:05Z

Does podman info work now, and if so, can you post it?

If it doesn't, that still tells us something

giuseppe · 2019-05-31T08:49:53Z

if you have no problems to run the container, could you run a process inside of it. And from the host do a cat /proc/$(PID_OF_THE_PROCESS_INSIDE_THE_CONTAINER)/uid_map?

You could for example do:

$ podman run --rm -d registry.hub.docker.com/library/ubuntu sleep 123.4
$ cat /proc/$(pgrep -f 123.4)/uid_map

panzerdev · 2019-06-04T18:14:58Z

@giuseppe

mpa@mpa-MacBookPro:~$ cat /proc/$(pgrep -f 123.4)/uid_map
         0       1000          1
         1     100000      65536

That's the result

@mheon

mpa@mpa-MacBookPro:~$ podman ps
CONTAINER ID  IMAGE                                          COMMAND      CREATED             STATUS                 PORTS  NAMES
4faddae85d25  registry.hub.docker.com/library/ubuntu:latest  sleep 123.4  About a minute ago  Up About a minute ago         determined_wozniak
mpa@mpa-MacBookPro:~$ podman exec -it determined_wozniak bash
exec failed: container_linux.go:346: starting container process caused "process_linux.go:101: executing setns process caused \"exit status 1\""
Error: exit status 1
mpa@mpa-MacBookPro:~$ podman info
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100 pc=0x1131cb3]

goroutine 1 [running]:
github.com/containers/libpod/libpod.(*Runtime).storeInfo(0xc4203ec1e0, 0x2461f78, 0x0, 0x0)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/libpod/info.go:119 +0x53
github.com/containers/libpod/libpod.(*Runtime).Info(0xc4203ec1e0, 0x16d3998, 0xc4203ec1e0, 0xc420911100, 0x0, 0x0)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/libpod/runtime.go:1114 +0x10d
main.infoCmd(0x243f7c0, 0x0, 0x0)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/info.go:60 +0xe8
main.glob..func64(0x23dcc60, 0x2461f78, 0x0, 0x0, 0x0, 0x0)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/info.go:33 +0x87
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x23dcc60, 0xc42000c070, 0x0, 0x0, 0x23dcc60, 0xc42000c070)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x23de420, 0x6, 0x24639a0, 0xc42019cf00)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(0x23de420, 0xc420712160, 0xc4200aa058)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/main.go:142 +0x3e
mpa@mpa-MacBookPro:~$

That's the result of podman info after I tried call exec.

Before:

mpa@mpa-MacBookPro:~$ podman info
host:
  BuildahVersion: 1.8.2
  Conmon:
    package: 'conmon: /usr/libexec/crio/conmon'
    path: /usr/libexec/crio/conmon
    version: 'conmon version , commit: '
  Distribution:
    distribution: ubuntu
    version: "16.04"
  MemFree: 5928730624
  MemTotal: 16696877056
  OCIRuntime:
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  SwapFree: 17024049152
  SwapTotal: 17049841664
  arch: amd64
  cpus: 8
  hostname: mpa-MacBookPro
  kernel: 4.4.0-148-generic
  os: linux
  rootless: true
  uptime: 26h 47m 40.53s (Approximately 1.08 days)
registries:
  blocked: null
  insecure: null
  search: null
store:
  ConfigFile: /home/mpa/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/mpa/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 1
  RunRoot: /tmp/1000
  VolumePath: /home/mpa/.local/share/containers/storage/volumes

giuseppe · 2019-06-06T06:52:21Z

could you please attach the output of podman --log-level=debug info --debug?

panzerdev · 2019-06-06T09:50:58Z

Of course @giuseppe

INFO[0000] running as rootless                          
DEBU[0000] Initializing boltdb state at /home/mpa/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver vfs                       
DEBU[0000] Using graph root /home/mpa/.local/share/containers/storage 
DEBU[0000] Using run root /tmp/1000                     
DEBU[0000] Using static dir /home/mpa/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/mpa/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Not configuring container store              
DEBU[0000] Initializing event backend file              
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x100 pc=0x1131cb3]

goroutine 1 [running]:
github.com/containers/libpod/libpod.(*Runtime).storeInfo(0xc4203d61e0, 0x2461f78, 0x0, 0x0)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/libpod/info.go:119 +0x53
github.com/containers/libpod/libpod.(*Runtime).Info(0xc4203d61e0, 0x16d3998, 0xc4203d61e0, 0xc420587200, 0x0, 0x0)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/libpod/runtime.go:1114 +0x10d
main.infoCmd(0x243f7c0, 0x0, 0x0)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/info.go:60 +0xe8
main.glob..func64(0x23dcc60, 0xc4202da010, 0x0, 0x1, 0x0, 0x0)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/info.go:33 +0x87
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x23dcc60, 0xc42003a130, 0x1, 0x1, 0x23dcc60, 0xc42003a130)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:762 +0x468
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x23de420, 0x6, 0x24639a0, 0xc42019af00)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x30a
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(0x23de420, 0xc420702160, 0xc4200ac058)
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
	/build/podman-qFWEMp/podman-1.3.2/src/github.com/containers/libpod/cmd/podman/main.go:142 +0x3e

giuseppe · 2019-06-06T10:43:03Z

thanks.

Something strange is happening here, I see "Not configuring container store"

it seems it is trying to use info without re-execing in the user namespace. That should never happen with info

on old kernels the ioctl NS_GET_PARENT is not available. Handle the error code and immediately return the same fd. It should be fine now that we use the namespace resolution using the conmon pid, so the namespace parent resolution is just a safety measure. Closes: containers#2968 Signed-off-by: Giuseppe Scrivano <[email protected]>

giuseppe · 2019-06-06T13:34:57Z

I see the issue, I've opened a PR here: #3271

openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Apr 18, 2019

rhatdan assigned baude Apr 18, 2019

mheon added the rootless label Apr 18, 2019

rhatdan assigned giuseppe Apr 18, 2019

giuseppe mentioned this issue Jun 6, 2019

rootless: skip NS_GET_PARENT on old kernels #3271

Merged

openshift-merge-robot closed this as completed in #3271 Jun 7, 2019

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 24, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman exec fails #2968

podman exec fails #2968

panzerdev commented Apr 18, 2019

rhatdan commented Apr 18, 2019

panzerdev commented Apr 18, 2019

mheon commented Apr 18, 2019

mheon commented Apr 18, 2019

panzerdev commented Apr 18, 2019 •

edited

Loading

mheon commented Apr 18, 2019

baude commented May 29, 2019

panzerdev commented May 30, 2019 •

edited

Loading

mheon commented May 31, 2019

giuseppe commented May 31, 2019

panzerdev commented Jun 4, 2019

giuseppe commented Jun 6, 2019

panzerdev commented Jun 6, 2019

giuseppe commented Jun 6, 2019

giuseppe commented Jun 6, 2019

podman exec fails #2968

podman exec fails #2968

Comments

panzerdev commented Apr 18, 2019

rhatdan commented Apr 18, 2019

panzerdev commented Apr 18, 2019

mheon commented Apr 18, 2019

mheon commented Apr 18, 2019

panzerdev commented Apr 18, 2019 • edited Loading

mheon commented Apr 18, 2019

baude commented May 29, 2019

panzerdev commented May 30, 2019 • edited Loading

mheon commented May 31, 2019

giuseppe commented May 31, 2019

panzerdev commented Jun 4, 2019

giuseppe commented Jun 6, 2019

panzerdev commented Jun 6, 2019

giuseppe commented Jun 6, 2019

giuseppe commented Jun 6, 2019

panzerdev commented Apr 18, 2019 •

edited

Loading

panzerdev commented May 30, 2019 •

edited

Loading