Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

go runtime panic of trying to re-add container with rootless pod #2964

Closed
Fodoj opened this issue Apr 17, 2019 · 15 comments · Fixed by #2967
Closed

go runtime panic of trying to re-add container with rootless pod #2964

Fodoj opened this issue Apr 17, 2019 · 15 comments · Fixed by #2967
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. rootless

Comments

@Fodoj
Copy link
Contributor

Fodoj commented Apr 17, 2019

/kind bug

Version

Podman 1.3.0-dev

OS

Fedora Workstation 29, Linux linux.fritz.box 5.0.4-200.fc29.x86_64 #1 SMP Mon Mar 25 02:27:33 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

How to reproduce

Note that probably not all ports are needed or even matter. But I still provide full commands I executed:

podman pod create --name mkdev-dev -p 5432 -p 8000 -p 3000 -p 6379
podman run -e POSTGRES_PASSWORD=secrets -v $(pwd)/tmp:/var/run/postgresql:z --pod mkdev-dev postgres:9.6.9 

Now postgres is running and attached to current session. Then I cancel it with CTRL+C and run the same podman run -e POSTGRESS_PASWORD ... command again.

Result:

runtime: epollwait on fd 4 failed with 9
fatal error: runtime: netpoll failed

runtime stack:
runtime.throw(0x169c8bb, 0x17)
	/usr/lib/golang/src/runtime/panic.go:608 +0x72
runtime.netpoll(0xf45c19161200, 0x0)
	/usr/lib/golang/src/runtime/netpoll_epoll.go:75 +0x215
runtime.sysmon()
	/usr/lib/golang/src/runtime/proc.go:4395 +0x41b
runtime.mstart1()
	/usr/lib/golang/src/runtime/proc.go:1282 +0xe6
runtime.mstart()
	/usr/lib/golang/src/runtime/proc.go:1248 +0x6e

goroutine 1 [syscall]:
github.com/containers/libpod/pkg/rootless._Cfunc_reexec_in_user_namespace_wait(0x700002951, 0x0)
	_cgo_gotypes.go:56 +0x4d
github.com/containers/libpod/pkg/rootless.JoinUserAndMountNS(0x2765, 0x0, 0x0, 0x0, 0x0)
	/home/fodoj/work/go/src/github.com/containers/libpod/pkg/rootless/rootless_linux.go:197 +0x2b4
main.setupRootless(0x268f5a0, 0xc000806380, 0x1, 0x7, 0x0, 0x0)
	/home/fodoj/work/go/src/github.com/containers/libpod/cmd/podman/main_local.go:142 +0x2cc
main.before(0x268f5a0, 0xc000806380, 0x1, 0x7, 0x1, 0xc0000d6618)
	/home/fodoj/work/go/src/github.com/containers/libpod/cmd/podman/main.go:101 +0x75
main.glob..func56(0x268f5a0, 0xc000806380, 0x1, 0x7, 0x0, 0x0)
	/home/fodoj/work/go/src/github.com/containers/libpod/cmd/podman/main.go:70 +0x49
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x268f5a0, 0xc0000dc020, 0x7, 0x7, 0x268f5a0, 0xc0000dc020)
	/home/fodoj/work/go/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:741 +0x584
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x2694b20, 0x6, 0x2719c60, 0xc0001bb200)
	/home/fodoj/work/go/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x2fd
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(0x2694b20, 0xc0006f2370, 0xc0000bc058)
	/home/fodoj/work/go/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800 +0x2b
main.main()
	/home/fodoj/work/go/src/github.com/containers/libpod/cmd/podman/main.go:149 +0x3c

goroutine 19 [syscall]:
os/signal.signal_recv(0x0)
	/usr/lib/golang/src/runtime/sigqueue.go:139 +0x9c
os/signal.loop()
	/usr/lib/golang/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.init.0
	/usr/lib/golang/src/os/signal/signal_unix.go:29 +0x41

goroutine 22 [chan receive]:
github.com/containers/libpod/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x26f1680)
	/home/fodoj/work/go/src/github.com/containers/libpod/vendor/github.com/golang/glog/glog.go:882 +0x8b
created by github.com/containers/libpod/vendor/github.com/golang/glog.init.0
	/home/fodoj/work/go/src/github.com/containers/libpod/vendor/github.com/golang/glog/glog.go:410 +0x203
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Apr 17, 2019
@Fodoj
Copy link
Contributor Author

Fodoj commented Apr 17, 2019

To be more prices, every command returns similar go panic. The only way to fix it is to find podman process (ps aux | grep podman) and kill the process. Then commands can be executed. But it also results in some bad corrupted state, because running exactly same podman run command returns this:

podman run -e POSTGRES_PASSWORD=secrets -v $(pwd)/tmp:/var/run/postgresql:z -d --pod mkdev-dev postgres:9.6.9 
Error: must provide image ID and image name to use an image: invalid argument

@Fodoj
Copy link
Contributor Author

Fodoj commented Apr 17, 2019

More info: I also tried removing some of resulting stopped containers and it returns a different go panic:

➜  application git:(improve-dev-env) ✗ podman ps --all
CONTAINER ID  IMAGE                             COMMAND               CREATED         STATUS                    PORTS                   NAMES
f303c43e9853  docker.io/library/postgres:9.6.9  docker-entrypoint...  11 minutes ago  Exited (0) 9 minutes ago  0.0.0.0:3000->3000/tcp  recursing_babbage
2b537bdd9edd  k8s.gcr.io/pause:3.1                                    11 minutes ago  Up 11 minutes ago         0.0.0.0:3000->3000/tcp  b8459bf2ead4-infra
➜  application git:(improve-dev-env) ✗ podman rm recursing_babbage
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1185951]

goroutine 50 [running]:
github.com/containers/libpod/libpod.(*storageService).DeleteContainer(0xc00086ce30, 0xc000714700, 0x40, 0x0, 0x92)
	/home/fodoj/work/go/src/github.com/containers/libpod/libpod/storage.go:180 +0x41
github.com/containers/libpod/libpod.(*Container).teardownStorage(0xc000942180, 0x18a35a0, 0xc000042268)
	/home/fodoj/work/go/src/github.com/containers/libpod/libpod/container_internal.go:350 +0x218
github.com/containers/libpod/libpod.(*Runtime).removeContainer(0xc0009662a0, 0x18a35a0, 0xc000042268, 0xc000942180, 0xc0004b0000, 0x0, 0x0)
	/home/fodoj/work/go/src/github.com/containers/libpod/libpod/runtime_ctr.go:382 +0x48d
github.com/containers/libpod/libpod.(*Runtime).RemoveContainer(0xc0009662a0, 0x18a35a0, 0xc000042268, 0xc000942180, 0xc0008e0000, 0x0, 0x0)
	/home/fodoj/work/go/src/github.com/containers/libpod/libpod/runtime_ctr.go:241 +0xa9
github.com/containers/libpod/pkg/adapter.(*LocalRuntime).RemoveContainers.func1(0xc0009422a0, 0xc0004b1728)
	/home/fodoj/work/go/src/github.com/containers/libpod/pkg/adapter/containers.go:162 +0x7a
github.com/containers/libpod/cmd/podman/shared.(*Pool).newWorker(0xc00041b640, 0x0)
	/home/fodoj/work/go/src/github.com/containers/libpod/cmd/podman/shared/workers.go:98 +0xf4
created by github.com/containers/libpod/cmd/podman/shared.(*Pool).Run
	/home/fodoj/work/go/src/github.com/containers/libpod/cmd/podman/shared/workers.go:72 +0x88

The only way to fix it was to run sudo rm -rf ~/.local/share/containers, which if course just wipes all the images and containers from my system.

@mheon
Copy link
Member

mheon commented Apr 17, 2019

@giuseppe PTAL - potentially related to rootless userns?

@Fodoj
Copy link
Contributor Author

Fodoj commented Apr 17, 2019

I found an easier way to re-produce. Just run this with Podman 1.3.0-dev:

podman pod create --name mkdev-dev -p 5432 -p 8000 -p 3000 -p 6379
podman run -d --pod mkdev-dev postgres:9.6.9
podman ps

And you will get same go panic with netpoll failed.

@baude
Copy link
Member

baude commented Apr 17, 2019

@Fodoj I think you have something wrong with your environment or podmans. I ran both the origina replicator and the one just above as both root and rootless. all of it worked as expected. I would recommend joining us on IRC to investigate further.

@Fodoj
Copy link
Contributor Author

Fodoj commented Apr 17, 2019

@baude sure, how do I do that?

@baude
Copy link
Member

baude commented Apr 17, 2019

we are on irc.freenode.net in #podman ...

@mheon mheon added the rootless label Apr 17, 2019
@mheon
Copy link
Member

mheon commented Apr 17, 2019

@giuseppe PTAL. From the stacktraces, it seems like a runtime panic out of Go while libpod is in the C code for joining a user namespace.

@Fodoj
Copy link
Contributor Author

Fodoj commented Apr 17, 2019

I downloaded latst build from here (no idea if that's correct place, but I couldnt find any other compiled version of master branch): https://apps.fedoraproject.org/packages/podman/overview/ and it also throws at me the panic, even though it also shows the result of command, for example:

runtime: epollwait on fd 4 failed with 9
fatal error: runtime: netpoll failed

runtime stack:
runtime.throw(0x565334a7b2d3, 0x17)
	/usr/lib/golang/src/runtime/panic.go:617 +0x74 fp=0x7f1bbe69d580 sp=0x7f1bbe69d550 pc=0x565333adfb24
runtime.netpoll(0x50d8a071400, 0x0)
	/usr/lib/golang/src/runtime/netpoll_epoll.go:75 +0x20b fp=0x7f1bbe69dbc8 sp=0x7f1bbe69d580 pc=0x565333add86b
runtime.sysmon()
	/usr/lib/golang/src/runtime/proc.go:4326 +0x376 fp=0x7f1bbe69dc58 sp=0x7f1bbe69dbc8 pc=0x565333aebf06
runtime.mstart1()
	/usr/lib/golang/src/runtime/proc.go:1206 +0xc8 fp=0x7f1bbe69dc80 sp=0x7f1bbe69dc58 pc=0x565333ae42b8
runtime.mstart()
	/usr/lib/golang/src/runtime/proc.go:1172 +0x71 fp=0x7f1bbe69dca8 sp=0x7f1bbe69dc80 pc=0x565333ae41d1

goroutine 1 [syscall]:
runtime.cgocall(0x565334a579d0, 0xc0008bfa60, 0xc000000000)
	/usr/lib/golang/src/runtime/cgocall.go:128 +0x5d fp=0xc0008bfa30 sp=0xc0008bf9f8 pc=0x565333ab62cd
github.com/containers/libpod/pkg/rootless._Cfunc_reexec_in_user_namespace_wait(0x700003062, 0xc000000000)
	_cgo_gotypes.go:56 +0x4b fp=0xc0008bfa60 sp=0xc0008bfa30 pc=0x565334663cfb
github.com/containers/libpod/pkg/rootless.JoinUserAndMountNS(0x24ec, 0x0, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/libpod-a87cf6fef8f84ded410d547aa772b8f942c5f273/_build/src/github.com/containers/libpod/pkg/rootless/rootless_linux.go:197 +0x2c6 fp=0xc0008bfae0 sp=0xc0008bfa60 pc=0x565334664f86
main.setupRootless(0x56533617f000, 0x565336209498, 0x0, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/libpod-a87cf6fef8f84ded410d547aa772b8f942c5f273/_build/src/github.com/containers/libpod/cmd/podman/main_local.go:142 +0x2a0 fp=0xc0008bfce0 sp=0xc0008bfae0 pc=0x565334a11920
main.before(0x56533617f000, 0x565336209498, 0x0, 0x0, 0x1, 0xc000010528)
	/builddir/build/BUILD/libpod-a87cf6fef8f84ded410d547aa772b8f942c5f273/_build/src/github.com/containers/libpod/cmd/podman/main.go:101 +0x77 fp=0xc0008bfd58 sp=0xc0008bfce0 pc=0x565334a105e7
main.glob..func56(0x56533617f000, 0x565336209498, 0x0, 0x0, 0x0, 0x0)
	/builddir/build/BUILD/libpod-a87cf6fef8f84ded410d547aa772b8f942c5f273/_build/src/github.com/containers/libpod/cmd/podman/main.go:70 +0x4b fp=0xc0008bfd98 sp=0xc0008bfd58 pc=0x565334a4bf6b
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x56533617f000, 0xc0000ce030, 0x0, 0x0, 0x56533617f000, 0xc0000ce030)
	/builddir/build/BUILD/libpod-a87cf6fef8f84ded410d547aa772b8f942c5f273/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:741 +0x572 fp=0xc0008bfe80 sp=0xc0008bfd98 pc=0x565333c88f82
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x565336187040, 0xc0000c6cc0, 0x7ffe2e2c8b9f, 0xf)
	/builddir/build/BUILD/libpod-a87cf6fef8f84ded410d547aa772b8f942c5f273/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:852 +0x2ee fp=0xc0008bff50 sp=0xc0008bfe80 pc=0x565333c8991e
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/builddir/build/BUILD/libpod-a87cf6fef8f84ded410d547aa772b8f942c5f273/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:800
main.main()
	/builddir/build/BUILD/libpod-a87cf6fef8f84ded410d547aa772b8f942c5f273/_build/src/github.com/containers/libpod/cmd/podman/main.go:149 +0x8a fp=0xc0008bff98 sp=0xc0008bff50 pc=0x565334a1095a
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:200 +0x214 fp=0xc0008bffe0 sp=0xc0008bff98 pc=0x565333ae14c4
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0008bffe8 sp=0xc0008bffe0 pc=0x565333b0d451

goroutine 2 [force gc (idle)]:
runtime.gopark(0x5653352b5c98, 0x5653361e24b0, 0x1410, 0x1)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc000078fb0 sp=0xc000078f90 pc=0x565333ae18c5
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:307
runtime.forcegchelper()
	/usr/lib/golang/src/runtime/proc.go:250 +0xbb fp=0xc000078fe0 sp=0xc000078fb0 pc=0x565333ae175b
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000078fe8 sp=0xc000078fe0 pc=0x565333b0d451
created by runtime.init.6
	/usr/lib/golang/src/runtime/proc.go:239 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x5653352b5c98, 0x5653361e2a60, 0x140c, 0x1)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc0000797a8 sp=0xc000079788 pc=0x565333ae18c5
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:307
runtime.bgsweep(0xc0000a0000)
	/usr/lib/golang/src/runtime/mgcsweep.go:89 +0x138 fp=0xc0000797d8 sp=0xc0000797a8 pc=0x565333ad4c78
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000797e0 sp=0xc0000797d8 pc=0x565333b0d451
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:208 +0x5a

goroutine 18 [finalizer wait]:
runtime.gopark(0x5653352b5c98, 0x5653362093f8, 0xc00007140f, 0x1)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc000078758 sp=0xc000078738 pc=0x565333ae18c5
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:307
runtime.runfinq()
	/usr/lib/golang/src/runtime/mfinal.go:175 +0xaa fp=0xc0000787e0 sp=0xc000078758 pc=0x565333acb7ba
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000787e8 sp=0xc0000787e0 pc=0x565333b0d451
created by runtime.createfing
	/usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 19 [timer goroutine (idle)]:
runtime.gopark(0x5653352b5c98, 0x5653361f07c0, 0x1414, 0x1)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc000074760 sp=0xc000074740 pc=0x565333ae18c5
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:307
runtime.timerproc(0x5653361f07c0)
	/usr/lib/golang/src/runtime/time.go:303 +0x277 fp=0xc0000747d8 sp=0xc000074760 pc=0x565333afe797
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000747e0 sp=0xc0000747d8 pc=0x565333b0d451
created by runtime.(*timersBucket).addtimerLocked
	/usr/lib/golang/src/runtime/time.go:169 +0x110

goroutine 20 [syscall]:
runtime.notetsleepg(0x565336209b00, 0xffffffffffffffff, 0x0)
	/usr/lib/golang/src/runtime/lock_futex.go:227 +0x38 fp=0xc000074f98 sp=0xc000074f68 pc=0x565333abdf38
os/signal.signal_recv(0x0)
	/usr/lib/golang/src/runtime/sigqueue.go:139 +0x9e fp=0xc000074fc0 sp=0xc000074f98 pc=0x565333af60ce
os/signal.loop()
	/usr/lib/golang/src/os/signal/signal_unix.go:23 +0x24 fp=0xc000074fe0 sp=0xc000074fc0 pc=0x565334043304
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000074fe8 sp=0xc000074fe0 pc=0x565333b0d451
created by os/signal.init.0
	/usr/lib/golang/src/os/signal/signal_unix.go:29 +0x43

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x5653352b5b30, 0xc00048f310, 0x1417, 0x0)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc000075760 sp=0xc000075740 pc=0x565333ae18c5
runtime.gcBgMarkWorker(0xc000058000)
	/usr/lib/golang/src/runtime/mgc.go:1836 +0x105 fp=0xc0000757d8 sp=0xc000075760 pc=0x565333acf2d5
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000757e0 sp=0xc0000757d8 pc=0x565333b0d451
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1784 +0x79

goroutine 22 [GC worker (idle)]:
runtime.gopark(0x5653352b5b30, 0xc00048f320, 0x1417, 0x0)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc000075f60 sp=0xc000075f40 pc=0x565333ae18c5
runtime.gcBgMarkWorker(0xc00005a500)
	/usr/lib/golang/src/runtime/mgc.go:1836 +0x105 fp=0xc000075fd8 sp=0xc000075f60 pc=0x565333acf2d5
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000075fe0 sp=0xc000075fd8 pc=0x565333b0d451
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1784 +0x79

goroutine 23 [GC worker (idle)]:
runtime.gopark(0x5653352b5b30, 0xc0004a6000, 0x1417, 0x0)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc000076760 sp=0xc000076740 pc=0x565333ae18c5
runtime.gcBgMarkWorker(0xc00005ca00)
	/usr/lib/golang/src/runtime/mgc.go:1836 +0x105 fp=0xc0000767d8 sp=0xc000076760 pc=0x565333acf2d5
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0000767e0 sp=0xc0000767d8 pc=0x565333b0d451
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1784 +0x79

goroutine 4 [GC worker (idle)]:
runtime.gopark(0x5653352b5b30, 0xc000042230, 0x1417, 0x0)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc000079f60 sp=0xc000079f40 pc=0x565333ae18c5
runtime.gcBgMarkWorker(0xc00005ef00)
	/usr/lib/golang/src/runtime/mgc.go:1836 +0x105 fp=0xc000079fd8 sp=0xc000079f60 pc=0x565333acf2d5
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000079fe0 sp=0xc000079fd8 pc=0x565333b0d451
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1784 +0x79

goroutine 5 [GC worker (idle)]:
runtime.gopark(0x5653352b5b30, 0xc000042240, 0x1417, 0x0)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc00007a760 sp=0xc00007a740 pc=0x565333ae18c5
runtime.gcBgMarkWorker(0xc000061400)
	/usr/lib/golang/src/runtime/mgc.go:1836 +0x105 fp=0xc00007a7d8 sp=0xc00007a760 pc=0x565333acf2d5
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00007a7e0 sp=0xc00007a7d8 pc=0x565333b0d451
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1784 +0x79

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x5653352b5b30, 0xc000042250, 0x1417, 0x0)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc00007af60 sp=0xc00007af40 pc=0x565333ae18c5
runtime.gcBgMarkWorker(0xc000063900)
	/usr/lib/golang/src/runtime/mgc.go:1836 +0x105 fp=0xc00007afd8 sp=0xc00007af60 pc=0x565333acf2d5
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00007afe0 sp=0xc00007afd8 pc=0x565333b0d451
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1784 +0x79

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x5653352b5b30, 0xc000042260, 0x1417, 0x0)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc00007b760 sp=0xc00007b740 pc=0x565333ae18c5
runtime.gcBgMarkWorker(0xc000066000)
	/usr/lib/golang/src/runtime/mgc.go:1836 +0x105 fp=0xc00007b7d8 sp=0xc00007b760 pc=0x565333acf2d5
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00007b7e0 sp=0xc00007b7d8 pc=0x565333b0d451
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1784 +0x79

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x5653352b5b30, 0xc000042270, 0x1417, 0x0)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc00007bf60 sp=0xc00007bf40 pc=0x565333ae18c5
runtime.gcBgMarkWorker(0xc000068500)
	/usr/lib/golang/src/runtime/mgc.go:1836 +0x105 fp=0xc00007bfd8 sp=0xc00007bf60 pc=0x565333acf2d5
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00007bfe0 sp=0xc00007bfd8 pc=0x565333b0d451
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1784 +0x79

goroutine 9 [chan receive]:
runtime.gopark(0x5653352b5c98, 0xc0000d8298, 0x170d, 0x3)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc0005d5ed0 sp=0xc0005d5eb0 pc=0x565333ae18c5
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:307
runtime.chanrecv(0xc0000d8240, 0xc0005d5fb0, 0xc0000de001, 0xc0000d8240)
	/usr/lib/golang/src/runtime/chan.go:524 +0x2ee fp=0xc0005d5f60 sp=0xc0005d5ed0 pc=0x565333ab901e
runtime.chanrecv2(0xc0000d8240, 0xc0005d5fb0, 0x0)
	/usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc0005d5f90 sp=0xc0005d5f60 pc=0x565333ab8d1b
github.com/containers/libpod/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x5653361e3c60)
	/builddir/build/BUILD/libpod-a87cf6fef8f84ded410d547aa772b8f942c5f273/_build/src/github.com/containers/libpod/vendor/github.com/golang/glog/glog.go:882 +0x8d fp=0xc0005d5fd8 sp=0xc0005d5f90 pc=0x5653343beead
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0005d5fe0 sp=0xc0005d5fd8 pc=0x565333b0d451
created by github.com/containers/libpod/vendor/github.com/golang/glog.init.0
	/builddir/build/BUILD/libpod-a87cf6fef8f84ded410d547aa772b8f942c5f273/_build/src/github.com/containers/libpod/vendor/github.com/golang/glog/glog.go:410 +0x274

goroutine 34 [syscall]:
runtime.notetsleepg(0x5653361f0a60, 0x6fc23859c, 0x2)
	/usr/lib/golang/src/runtime/lock_futex.go:227 +0x38 fp=0xc0005d1f60 sp=0xc0005d1f30 pc=0x565333abdf38
runtime.timerproc(0x5653361f0a40)
	/usr/lib/golang/src/runtime/time.go:311 +0x2ee fp=0xc0005d1fd8 sp=0xc0005d1f60 pc=0x565333afe80e
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0005d1fe0 sp=0xc0005d1fd8 pc=0x565333b0d451
created by runtime.(*timersBucket).addtimerLocked
	/usr/lib/golang/src/runtime/time.go:169 +0x110
REPOSITORY                   TAG     IMAGE ID       CREATED         SIZE
docker.io/library/postgres   9.6.9   106bdfb062df   8 months ago    243 MB
k8s.gcr.io/pause             3.1     da86e6ba6ca1   16 months ago   747 kB
[1]    12374 abort (core dumped)  /usr/bin/podman images

So, bug is not reproducible on 1.2.0, but I can easily reproduce it if I compile master branch myself or if I download the latest rpm from here https://apps.fedoraproject.org/packages/podman/overview/.

@mheon
Copy link
Member

mheon commented Apr 17, 2019

1.2 is missing the refactor of rootless user namespaces, so this supports my theory

@giuseppe
Copy link
Member

I think this is caused by us closing the fds not used by Podman. Looks like the detection for how many fds are open is detecting some files that are later closed and we are in facts closing some fds used by golang.

To reproduce you'll need to open some FDs like:

touch /tmp/foo
exec 20<> /tmp/foo
exec 21<> /tmp/foo
exec 22<> /tmp/foo
exec 23<> /tmp/foo

PR here: #2967

giuseppe added a commit to giuseppe/libpod that referenced this issue Apr 18, 2019
we were previously closing as many FDs as they were open when we first
started Podman in the range (3-MAX-FD).  This would cause issues if
there were empty intervals, as these FDs are later on used by the
Golang runtime.  Store exactly what FDs were first open in a fd_set,
so that we can close exactly the FDs that were open at startup.

Closes: containers#2964

Signed-off-by: Giuseppe Scrivano <[email protected]>
@Fodoj
Copy link
Contributor Author

Fodoj commented Apr 19, 2019

With latest master branch error looks different:

➜  application git:(improve-dev-env) ✗ podman pod create --name mkdev-dev -p 5432 -p 8000 -p 3000 -p 6379
57fb3fe63f368744c0e831d9f5ab3baf8305c94b9f7405edc1dc50f8129349f0
➜  application git:(improve-dev-env) ✗ podman ps --all                                                   
CONTAINER ID  IMAGE                 COMMAND  CREATED         STATUS   PORTS                   NAMES
106ce8e2be39  k8s.gcr.io/pause:3.1           21 seconds ago  Created  0.0.0.0:3000->3000/tcp  57fb3fe63f36-infra
➜  application git:(improve-dev-env) ✗ podman run -d --pod mkdev-dev postgres:9.6.9                      
Trying to pull docker.io/library/postgres:9.6.9...Getting image source signatures
Copying blob 0626c6149c90 done
Copying blob be8881be8156 done
Copying blob d45b5ac60cd5 done
Copying blob bcc05f43b4de done
Copying blob 67f823cf5f8b done
Copying blob e25dcd1f62ca done
Copying blob 78c4cc9b5f06 done
Copying blob b0c2f7780de2 done
Copying blob 1cd39ae8e6f8 done
Copying blob c3c9ac2352c5 done
Copying blob 3945b9fc0c53 done
Copying blob 2bff6467ae03 done
Copying blob 0c30f6cc3676 done
Copying blob e3e21dfd259c done
Copying config 106bdfb062 done
Writing manifest to image destination
Storing signatures
ERRO[0053] error starting some container dependencies   
ERRO[0053] "error from slirp4netns while setting up port redirection: map[desc:bad request: add_hostfwd: slirp_add_hostfwd failed]" 
Error: error starting some containers: internal libpod error

@Fodoj
Copy link
Contributor Author

Fodoj commented Apr 19, 2019

@giuseppe @mheon

@giuseppe
Copy link
Member

that is a different issue, caused by slirp4netns. What version of slirp4netns are you using?

@Fodoj
Copy link
Contributor Author

Fodoj commented Apr 19, 2019

@giuseppe after doing complete clean and reboot (and also noticing that there was dangling slirp4netns process on port 8000) (and also disabling and stopping docker daemon) it worked fine. Error message is not that clear though. So yes, different issue and a result of local inconsistent setup..

Installed Packages
Name         : slirp4netns
Version      : 0.3
Release      : 0.alpha.2.git30883b5.fc29
Architecture : x86_64
Size         : 135 k
Source       : slirp4netns-0.3-0.alpha.2.git30883b5.fc29.src.rpm
Repository   : @System
From repo    : updates
Summary      : slirp for network namespaces
URL          : https://github.com/rootless-containers/slirp4netns
License      : GPLv2
Description  : slirp for network namespaces, without copying buffers across the namespaces.

@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 24, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 24, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. rootless
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants