Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker Compose freezing with Podman during up/down #10922

Closed
djnotes opened this issue Jul 13, 2021 · 8 comments · Fixed by #10939
Closed

Docker Compose freezing with Podman during up/down #10922

djnotes opened this issue Jul 13, 2021 · 8 comments · Fixed by #10939
Assignees
Labels
In Progress This issue is actively being worked by the assignee, please do not work on this at this time. kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.

Comments

@djnotes
Copy link

djnotes commented Jul 13, 2021

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When starting a project with docker-compose (version 1.29.2), the command freezes and then gives a timeout. Podman socket is active (systemctl --user status podman.sock is OK)

  1. Start podman socket

  2. Run docker-compose up

Describe the results you received:

Creating myapp_redis_1 ... error
Creating myapp_db_1    ... 

ERROR: for myapp_redis_1  error opening "/run/user/1002/rootless-cni/rootless-cni.lck": no such file or directory

ERROR: for myapp_db_1  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)

ERROR: for redis  error opening "/run/user/1002/rootless-cni/rootless-cni.lck": no such file or directory

ERROR: for db  UnixHTTPConnectionPool(host='localhost', port=None): Read timed out. (read timeout=60)
ERROR: An HTTP request took too long to complete. Retry with --verbose to obtain debug information.
If you encounter this issue regularly because of slow network conditions, consider setting COMPOSE_HTTP_TIMEOUT to a higher value (current value: 60).

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

3.2.2

Output of podman info --debug:

Takes very long to complete. Basically keeps freezing with no output. 

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.2-1.fc34.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

@openshift-ci openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Jul 13, 2021
@baude
Copy link
Member

baude commented Jul 13, 2021

docker-compose file?

@abustany
Copy link

abustany commented Jul 13, 2021

same issue here (same distro/podman version), here is the file:

version: "3.7"
services:
  db:
    container_name: postgres
    restart: always
    image: postgres:10.5-alpine
    environment:
      POSTGRES_USER: "user"
      POSTGRES_PASSWORD: "password" # TODO: Change this
      POSTGRES_DB: "user"
      MAX_CONNECTIONS: 500 # TODO: do not hard code use proper bounce
    volumes:
      - pgdata:/var/lib/postgresql/data
    ports:
      - "5432:5432"

  cache:
    container_name: cache
    restart: always
    image: redis:4.0-alpine
    ports:
      - "6379:6379"

volumes:
  pgdata:

@abustany
Copy link

Here are some logs that might be relevant:

Jul 13 15:12:06 adrien-xps13 podman[310658]: time="2021-07-13T15:12:06+02:00" level=warning msg="Recovering from API handler panic: error opening \"/run/user/1000/rootless-cni/rootless-cni.lck\": no such file or directory, goroutine 596 [running]:\ngithub.com/containers/podman/pkg/api/server.(*APIServer).APIHandler.func1.1(0x561399082710, 0xc000da0540)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/handler_api.go:24 +0xbd\npanic(0x561398d873a0, 0xc000841400)\n\t/usr/lib/golang/src/runtime/panic.go:965 +0x1b9\ngithub.com/containers/podman/vendor/github.com/containers/storage/pkg/lockfile.(*lockfile).lock(0xc0007a04b0, 0x561399000001)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/github.com/containers/storage/pkg/lockfile/lockfile_unix.go:107 +0x388\ngithub.com/containers/podman/vendor/github.com/containers/storage/pkg/lockfile.(*lockfile).Lock(0xc0007a04b0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/github.com/containers/storage/pkg/lockfile/lockfile_unix.go:129 +0x3f\ngithub.com/containers/podman/libpod.(*Runtime).GetRootlessCNINetNs(0xc0001e2fc0, 0xc000ac7a01, 0x0, 0x0, 0x0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/networking_linux.go:279 +0x228\ngithub.com/containers/podman/libpod.(*Runtime).setUpOCICNIPod(0xc0001e2fc0, 0xc000def220, 0xe, 0xc000def220, 0xe, 0xc000aa31c0, 0x40, 0xc000aa3700, 0x3d, 0xc0008adc00, ...)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/networking_linux.go:463 +0x50\ngithub.com/containers/podman/libpod.(*Container).NetworkConnect(0xc0007a8820, 0xc000aa30c0, 0x40, 0xc000def950, 0xe, 0xc000a3b0c0, 0x2, 0x4, 0x0, 0x0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/networking_linux.go:1168 +0x578\ngithub.com/containers/podman/libpod.(*Runtime).ConnectContainerToNetwork(0xc0001e2fc0, 0xc000aa30c0, 0x40, 0xc000aa3055, 0xe, 0xc000a3b0c0, 0x2, 0x4, 0xb200000000000000, 0xc000c11620)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/networking_linux.go:1228 +0xd1\ngithub.com/containers/podman/pkg/api/handlers/compat.Connect(0x561399082710, 0xc000da0540, 0xc0009e9a00)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/handlers/compat/networks.go:353 +0x1e5\ngithub.com/containers/podman/pkg/api/server.(*APIServer).APIHandler.func1.2(0x561399082710, 0xc000da0540, 0xc0009e9900)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/handler_api.go:66 +0x1004\ngithub.com/containers/podman/pkg/api/server.(*APIServer).APIHandler.func1(0x561399082710, 0xc000da0540, 0xc0009e9900)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/handler_api.go:69 +0xa8\nnet/http.HandlerFunc.ServeHTTP(0xc00000fe18, 0x561399082710, 0xc000da0540, 0xc0009e9900)\n\t/usr/lib/golang/src/net/http/server.go:2069 +0x46\ngithub.com/containers/podman/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00049e000, 0x561399082710, 0xc000da0540, 0xc0009e9700)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/github.com/gorilla/mux/mux.go:210 +0xd3\nnet/http.serverHandler.ServeHTTP(0xc00033a900, 0x561399082710, 0xc000da0540, 0xc0009e9700)\n\t/usr/lib/golang/src/net/http/server.go:2887 +0xa5\nnet/http.(*conn).serve(0xc0007a8780, 0x56139908cf08, 0xc000a3af00)\n\t/usr/lib/golang/src/net/http/server.go:1952 +0x8cd\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 1 [chan receive]:\ngithub.com/containers/podman/pkg/api/server.(*APIServer).Serve(0xc00033a900, 0x12a05f200, 0xc000118340)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/server.go:227 +0x174\ngithub.com/containers/podman/cmd/podman/system.restService(0x0, 0x0, 0x12a05f200, 0x561399b84220, 0xc0001bfe00, 0x561399c15900, 0x0, 0x0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/cmd/podman/system/service_abi.go:85 +0x25e\ngithub.com/containers/podman/cmd/podman/system.service(0x561399b84220, 0x561399c478b0, 0x0, 0x0, 0x0, 0x0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/cmd/podman/system/service.go:98 +0x1b2\ngithub.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).execute(0x561399b84220, 0xc000136030, 0x0, 0x0, 0x561399b84220, 0xc000136030)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:852 +0x472\ngithub.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x561399b87420, 0xc000130020, 0x561398e0c6e0, 0x561399c478b0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:960 +0x375\ngithub.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).Execute(...)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:897\ngithub.com/containers/podman/vendor/github.com/spf13/cobra.(*Command).ExecuteContext(...)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/github.com/spf13/cobra/command.go:890\nmain.Execute()\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/cmd/podman/root.go:90 +0xe9\nmain.main()\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/cmd/podman/main.go:38 +0x94\n\ngoroutine 52 [chan receive]:\ngithub.com/containers/podman/vendor/k8s.io/klog/v2.(*loggingT).flushDaemon(0x561399c15b00)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/k8s.io/klog/v2/klog.go:1164 +0x8d\ncreated by github.com/containers/podman/vendor/k8s.io/klog/v2.init.0\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/k8s.io/klog/v2/klog.go:418 +0xdf\n\ngoroutine 8 [syscall]:\nos/signal.signal_recv(0x0)\n\t/usr/lib/golang/src/runtime/sigqueue.go:168 +0xa5\nos/signal.loop()\n\t/usr/lib/golang/src/os/signal/signal_unix.go:23 +0x25\ncreated by os/signal.Notify.func1.1\n\t/usr/lib/golang/src/os/signal/signal.go:151 +0x46\n\ngoroutine 9 [select]:\ngithub.com/containers/podman/libpod/shutdown.Start.func1()\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:45 +0xab\ncreated by github.com/containers/podman/libpod/shutdown.Start\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/shutdown/handler.go:44 +0x116\n\ngoroutine 21 [chan receive]:\ngithub.com/containers/podman/pkg/domain/infra.StartWatcher.func1(0xc0006c8720, 0xc0001e2fc0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:369 +0x7c\ncreated by github.com/containers/podman/pkg/domain/infra.StartWatcher\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/domain/infra/runtime_libpod.go:365 +0xb7\n\ngoroutine 11 [sleep]:\ntime.Sleep(0x5f5e100)\n\t/usr/lib/golang/src/runtime/time.go:193 +0xd8\ngithub.com/containers/podman/libpod.(*Runtime).libimageEvents.func2(0xc0001e2fc0, 0x56139902bfe8)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/runtime.go:737 +0x299\ncreated by github.com/containers/podman/libpod.(*Runtime).libimageEvents\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/runtime.go:712 +0x4d\n\ngoroutine 12 [chan receive]:\ngithub.com/containers/podman/pkg/api/server.(*APIServer).Serve.func2(0xc00033a900)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/server.go:195 +0x48\ncreated by github.com/containers/podman/pkg/api/server.(*APIServer).Serve\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/server.go:194 +0xf1\n\ngoroutine 13 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092c3b0, 0x72, 0x0)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0006e0098, 0x72, 0x0, 0x0, 0x5613989ee19d)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Accept(0xc0006e0080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:401 +0x212\nnet.(*netFD).accept(0xc0006e0080, 0x561399bc4d08, 0x561398d867a0, 0x561399bc4c90)\n\t/usr/lib/golang/src/net/fd_unix.go:172 +0x45\nnet.(*UnixListener).accept(0xc000490000, 0xc000083e08, 0x5613989bc593, 0xc000490150)\n\t/usr/lib/golang/src/net/unixsock_posix.go:162 +0x34\nnet.(*UnixListener).Accept(0xc000490000, 0xc000083e58, 0x18, 0xc0002f7e00, 0x5613979c5aad)\n\t/usr/lib/golang/src/net/unixsock.go:260 +0x66\nnet/http.(*Server).Serve(0xc00033a900, 0x561399082530, 0xc000490000, 0x0, 0x0)\n\t/usr/lib/golang/src/net/http/server.go:2981 +0x292\ngithub.com/containers/podman/pkg/api/server.(*APIServer).Serve.func4(0xc00033a900, 0xc0000991a0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/server.go:219 +0x49\ncreated by github.com/containers/podman/pkg/api/server.(*APIServer).Serve\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/server.go:218 +0x154\n\ngoroutine 14 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092c2c8, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0006e0318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0006e0300, 0xc00002c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0006e0300, 0xc00002c000, 0x1000, 0x1000, 0x0, 0x7f572092c2d0, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0001328b0, 0xc00002c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc000312390, 0xc00002c000, 0x1000, 0x1000, 0xc033822c0a975654, 0x2591b6fb2, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc000099260)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc000099260, 0x4, 0x2591b6fb2, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc0005319a0, 0x56139908cf08, 0xc0004b9d80)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 382 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d6cc8, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0005f5118, 0x72, 0x2000, 0x2000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0005f5100, 0xc000d9c000, 0x2000, 0x2000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0005f5100, 0xc000d9c000, 0x2000, 0x2000, 0x0, 0xc0000a4db8, 0x5613976cbb8e)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0006b0508, 0xc000d9c000, 0x2000, 0x2000, 0xc000516e30, 0x561397693dd7, 0x561399c4c548)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\ngithub.com/containers/podman/libpod.httpAttachNonTerminalCopy(0xc0006b0508, 0xc000c2f2b0, 0xc000779140, 0x40, 0xc000010101, 0x1, 0x1)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:1600 +0x92\ngithub.com/containers/podman/libpod.(*ConmonOCIRuntime).HTTPAttach.func5(0xc00010e900, 0xc0007a80a0, 0xc000d65301, 0xc0006b0508, 0xc000c2f2b0, 0x101, 0xc00010e960)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:682 +0x26c\ncreated by github.com/containers/podman/libpod.(*ConmonOCIRuntime).HTTPAttach\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:667 +0x605\n\ngoroutine 28 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092c1e0, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0000ac518, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0000ac500, 0xc000762000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0000ac500, 0xc000762000, 0x1000, 0x1000, 0x0, 0x7f572092c1e8, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0006e8478, 0xc000762000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc0007199e0, 0xc000762000, 0x1000, 0x1000, 0xc033822c0e5da075, 0x25ce1b9db, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc0006c8e40)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc0006c8e40, 0x4, 0x25ce1b9db, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc000760000, 0x56139908cf08, 0xc00074ef40)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 154 [select]:\ngithub.com/containers/podman/libpod.(*ConmonOCIRuntime).HTTPAttach(0xc00028f9a0, 0xc0007a80a0, 0xc0009e8200, 0x561399082710, 0xc000028fc0, 0xc000991883, 0x0, 0x0, 0xc00082c2a0, 0xc0001e0001, ...)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:697 +0x6bb\ngithub.com/containers/podman/libpod.(*Container).HTTPAttach(0xc0007a80a0, 0xc0009e8200, 0x561399082710, 0xc000028fc0, 0xc000991883, 0x0, 0x0, 0x1, 0xc00082c2a0, 0x0, ...)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/container_api.go:308 +0x23a\ngithub.com/containers/podman/pkg/api/handlers/compat.AttachContainer(0x561399082710, 0xc000028fc0, 0xc0009e8200)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/handlers/compat/containers_attach.go:103 +0x56c\ngithub.com/containers/podman/pkg/api/server.(*APIServer).APIHandler.func1.2(0x561399082710, 0xc000028fc0, 0xc0009e8100)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/handler_api.go:66 +0x1004\ngithub.com/containers/podman/pkg/api/server.(*APIServer).APIHandler.func1(0x561399082710, 0xc000028fc0, 0xc0009e8100)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/handler_api.go:69 +0xa8\nnet/http.HandlerFunc.ServeHTTP(0xc00000e6d8, 0x561399082710, 0xc000028fc0, 0xc0009e8100)\n\t/usr/lib/golang/src/net/http/server.go:2069 +0x46\ngithub.com/containers/podman/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00049e000, 0x561399082710, 0xc000028fc0, 0xc00079df00)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/github.com/gorilla/mux/mux.go:210 +0xd3\nnet/http.serverHandler.ServeHTTP(0xc00033a900, 0x561399082710, 0xc000028fc0, 0xc00079df00)\n\t/usr/lib/golang/src/net/http/server.go:2887 +0xa5\nnet/http.(*conn).serve(0xc000549ea0, 0x56139908cf08, 0xc000137300)\n\t/usr/lib/golang/src/net/http/server.go:1952 +0x8cd\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 80 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092bc70, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc00097be98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc00097be80, 0xc00098e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc00097be80, 0xc00098e000, 0x1000, 0x1000, 0x0, 0x7f572092bc78, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc000133050, 0xc00098e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc00098b5c0, 0xc00098e000, 0x1000, 0x1000, 0xc033822c0f60ab31, 0x25de4c49a, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc000099800)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc000099800, 0x4, 0x25de4c49a, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc000548280, 0x56139908cf08, 0xc00097e400)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 135 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092b618, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000926298, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000926280, 0xc0008fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc000926280, 0xc0008fe000, 0x1000, 0x1000, 0x0, 0x7f572092b620, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc000132208, 0xc0008fe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc0004300f0, 0xc0008fe000, 0x1000, 0x1000, 0xc033822c2d2f9949, 0x27bb3b2a8, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc0005f03c0)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc0005f03c0, 0x4, 0x27bb3b2a8, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc000530280, 0x56139908cf08, 0xc0007d0540)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 119 [select]:\ngithub.com/containers/podman/libpod.(*ConmonOCIRuntime).HTTPAttach(0xc00028f9a0, 0xc0007605a0, 0xc0000dca00, 0x561399082710, 0xc0003262a0, 0xc000042d5b, 0x0, 0x0, 0xc00082c1c0, 0xc0001e0001, ...)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:697 +0x6bb\ngithub.com/containers/podman/libpod.(*Container).HTTPAttach(0xc0007605a0, 0xc0000dca00, 0x561399082710, 0xc0003262a0, 0xc000042d5b, 0x0, 0x0, 0x1, 0xc00082c1c0, 0x0, ...)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/container_api.go:308 +0x23a\ngithub.com/containers/podman/pkg/api/handlers/compat.AttachContainer(0x561399082710, 0xc0003262a0, 0xc0000dca00)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/handlers/compat/containers_attach.go:103 +0x56c\ngithub.com/containers/podman/pkg/api/server.(*APIServer).APIHandler.func1.2(0x561399082710, 0xc0003262a0, 0xc0000dc800)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/handler_api.go:66 +0x1004\ngithub.com/containers/podman/pkg/api/server.(*APIServer).APIHandler.func1(0x561399082710, 0xc0003262a0, 0xc0000dc800)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/handler_api.go:69 +0xa8\nnet/http.HandlerFunc.ServeHTTP(0xc00000e6d8, 0x561399082710, 0xc0003262a0, 0xc0000dc800)\n\t/usr/lib/golang/src/net/http/server.go:2069 +0x46\ngithub.com/containers/podman/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00049e000, 0x561399082710, 0xc0003262a0, 0xc0000dc600)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/github.com/gorilla/mux/mux.go:210 +0xd3\nnet/http.serverHandler.ServeHTTP(0xc00033a900, 0x561399082710, 0xc0003262a0, 0xc0000dc600)\n\t/usr/lib/golang/src/net/http/server.go:2887 +0xa5\nnet/http.(*conn).serve(0xc000760500, 0x56139908cf08, 0xc000912d00)\n\t/usr/lib/golang/src/net/http/server.go:1952 +0x8cd\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 131 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092b7e8, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000846398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000846380, 0xc0008c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc000846380, 0xc0008c1000, 0x1000, 0x1000, 0x0, 0x7f572092b7f0, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0006b0050, 0xc0008c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc00043c120, 0xc0008c1000, 0x1000, 0x1000, 0xc033822c1559f3e8, 0x263de0dc9, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc0002b3920)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc0002b3920, 0x4, 0x263de0dc9, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc000760320, 0x56139908cf08, 0xc0007d0000)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 39 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092bd58, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0008a6198, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0008a6180, 0xc00099e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0008a6180, 0xc00099e000, 0x1000, 0x1000, 0x0, 0x7f572092bd60, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc000010050, 0xc00099e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc00099a330, 0xc00099e000, 0x1000, 0x1000, 0xc033822c11c6c775, 0x2604ae0d8, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc000099a40)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc000099a40, 0x4, 0x2604ae0d8, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc000530460, 0x56139908cf08, 0xc00097e7c0)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 59 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092c0f8, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000645c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000645c80, 0xc0007ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc000645c80, 0xc0007ab000, 0x1000, 0x1000, 0x0, 0x7f572092c100, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0006b0120, 0xc0007ab000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc000643500, 0xc0007ab000, 0x1000, 0x1000, 0xc033822c0ecab8c4, 0x25d4ed22b, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc0003142a0)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc0003142a0, 0x4, 0x25d4ed22b, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc0007a8000, 0x56139908cf08, 0xc0007d0740)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 30 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092c010, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000544718, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000544700, 0xc000850000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc000544700, 0xc000850000, 0x1000, 0x1000, 0x0, 0x7f572092c018, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0006e8008, 0xc000850000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc0004900c0, 0xc000850000, 0x1000, 0x1000, 0xc033822c0e3d9c61, 0x25cc1b5bf, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc0006c8000)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc0006c8000, 0x4, 0x25cc1b5bf, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc000760140, 0x56139908cf08, 0xc00074ea40)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 98 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092bf28, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0003fd818, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0003fd800, 0xc0004ed000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0003fd800, 0xc0004ed000, 0x1000, 0x1000, 0x0, 0x7f572092bf30, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0006b0ea8, 0xc0004ed000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc0004a4300, 0xc0004ed000, 0x1000, 0x1000, 0xc033822c0e940af0, 0x25d182454, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc000315680)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc000315680, 0x4, 0x25d182454, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc0007a85a0, 0x56139908cf08, 0xc0004b8040)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 101 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092baa0, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000721318, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000721300, 0xc0008b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc000721300, 0xc0008b5000, 0x1000, 0x1000, 0x0, 0x7f572092baa8, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0006b1298, 0xc0008b5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc000329980, 0xc0008b5000, 0x1000, 0x1000, 0xc033822c12ce02b1, 0x261521c10, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc0008aa420)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc0008aa420, 0x4, 0x261521c10, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc0007a90e0, 0x56139908cf08, 0xc00003ca80)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 77 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092be40, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000170098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000170080, 0xc000900000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc000170080, 0xc000900000, 0x1000, 0x1000, 0x0, 0x7f572092be48, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc000132118, 0xc000900000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc00032bec0, 0xc000900000, 0x1000, 0x1000, 0xc033822c0f249def, 0x25da8b758, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc000098d80)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc000098d80, 0x4, 0x25da8b758, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc0006d14a0, 0x56139908cf08, 0xc00035c280)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 130 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092b8d0, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000846218, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000846200, 0xc000334000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc000846200, 0xc000334000, 0x1000, 0x1000, 0x0, 0x7f572092b8d8, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0006b0048, 0xc000334000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc00099a7e0, 0xc000334000, 0x1000, 0x1000, 0xc033822c16f7b922, 0x2657bd2a5, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc000314de0)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc000314de0, 0x4, 0x2657bd2a5, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc0007601e0, 0x56139908cf08, 0xc000136c40)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 84 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092b9b8, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000842098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000842080, 0xc000834000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc000842080, 0xc000834000, 0x1000, 0x1000, 0x0, 0x7f572092b9c0, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc000132018, 0xc000834000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc000838030, 0xc000834000, 0x1000, 0x1000, 0xc033822c12dc201b, 0x261603979, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc000098180)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc000098180, 0x4, 0x261603979, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc0007600a0, 0x56139908cf08, 0xc00074eb00)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 41 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092bb88, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0008a6398, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0008a6380, 0xc000833000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0008a6380, 0xc000833000, 0x1000, 0x1000, 0x0, 0x7f572092bb90, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc000010238, 0xc000833000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc000328c30, 0xc000833000, 0x1000, 0x1000, 0xc033822c11dd1ba5, 0x260613505, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc0008aa240)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc0008aa240, 0x4, 0x260613505, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc000530640, 0x56139908cf08, 0xc00003c340)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 133 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f572092b700, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000926118, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000926100, 0xc0008de000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc000926100, 0xc0008de000, 0x1000, 0x1000, 0x0, 0x7f572092b708, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0001321d0, 0xc0008de000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc00007e1e0, 0xc0008de000, 0x1000, 0x1000, 0xc033822c2d2f1650, 0x27bb32fae, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc0005f0240)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc0005f0240, 0x4, 0x27bb32fae, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc000530140, 0x56139908cf08, 0xc0007d03c0)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 485 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d74f0, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000c80198, 0x72, 0x2000, 0x2000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000c80180, 0xc000ade000, 0x2000, 0x2000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc000c80180, 0xc000ade000, 0x2000, 0x2000, 0x0, 0xc0000a6db8, 0x5613976cbb8e)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc000132050, 0xc000ade000, 0x2000, 0x2000, 0x203000, 0xc0007d9e00, 0xc0003e3628)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\ngithub.com/containers/podman/libpod.httpAttachNonTerminalCopy(0xc000132050, 0xc000b2e4f0, 0xc0003f6940, 0x40, 0xc000010101, 0x1, 0x1)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:1600 +0x92\ngithub.com/containers/podman/libpod.(*ConmonOCIRuntime).HTTPAttach.func5(0xc0004e4100, 0xc0007605a0, 0xc000abd301, 0xc000132050, 0xc000b2e4f0, 0x101, 0xc0004e4120)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:682 +0x26c\ncreated by github.com/containers/podman/libpod.(*ConmonOCIRuntime).HTTPAttach\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:667 +0x605\n\ngoroutine 383 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d7a60, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0009b9998, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0009b9980, 0xc000624000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0009b9980, 0xc000624000, 0x8000, 0x8000, 0x561399c4c548, 0xc0003dfda8, 0x561397688f2a)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc000011b08, 0xc000624000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc0009b7650, 0xc000624000, 0x8000, 0x8000, 0x8000, 0x561398d87660, 0x1)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).Read(0xc000099f20, 0xc000624000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/bufio/bufio.go:213 +0x144\ngithub.com/containers/podman/utils.CopyDetachable(0x561399059008, 0xc0006b0508, 0x561399056328, 0xc000c2f2b0, 0xc00040f090, 0x2, 0x8, 0xc000a00000, 0x1000, 0x1000)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/utils/utils.go:61 +0xa8\ngithub.com/containers/podman/libpod.(*ConmonOCIRuntime).HTTPAttach.func6(0xc0006b0508, 0xc000c2f2b0, 0xc00040f090, 0x2, 0x8, 0xc00010e9c0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:690 +0x88\ncreated by github.com/containers/podman/libpod.(*ConmonOCIRuntime).HTTPAttach\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:689 +0x7f7\n\ngoroutine 208 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d7978, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000c511b8, 0x72, 0x201, 0x200, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000c511a0, 0xc000396a00, 0x200, 0x200, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nos.(*File).read(...)\n\t/usr/lib/golang/src/os/file_posix.go:31\nos.(*File).Read(0xc0001323d0, 0xc000396a00, 0x200, 0x200, 0xc000c8f680, 0x5613976e7585, 0xc000b61b88)\n\t/usr/lib/golang/src/os/file.go:117 +0x85\nbytes.(*Buffer).ReadFrom(0xc0003bbb60, 0x5613990593a8, 0xc0001323d0, 0x7f5720083028, 0xc0003bbb60, 0xc000c8f701)\n\t/usr/lib/golang/src/bytes/buffer.go:204 +0xbe\nio.copyBuffer(0x561399056428, 0xc0003bbb60, 0x5613990593a8, 0xc0001323d0, 0x0, 0x0, 0x0, 0x561397680ef5, 0xc000b61c20, 0xc000c8f7b0)\n\t/usr/lib/golang/src/io/io.go:409 +0x357\nio.Copy(...)\n\t/usr/lib/golang/src/io/io.go:382\nos/exec.(*Cmd).writerDescriptor.func1(0xc000b61c20, 0xc000c8f7b0)\n\t/usr/lib/golang/src/os/exec/exec.go:311 +0x65\nos/exec.(*Cmd).Start.func1(0xc00072c9a0, 0xc000c58600)\n\t/usr/lib/golang/src/os/exec/exec.go:441 +0x29\ncreated by os/exec.(*Cmd).Start\n\t/usr/lib/golang/src/os/exec/exec.go:440 +0x630\n\ngoroutine 107 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d7320, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000c0d278, 0x72, 0x8001, 0x8000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000c0d260, 0xc000104000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nos.(*File).read(...)\n\t/usr/lib/golang/src/os/file_posix.go:31\nos.(*File).Read(0xc000010900, 0xc000104000, 0x8000, 0x8000, 0x52, 0x0, 0x0)\n\t/usr/lib/golang/src/os/file.go:117 +0x85\nio.copyBuffer(0x561399058e28, 0xc000a60a68, 0x5613990593a8, 0xc000010900, 0xc000104000, 0x8000, 0x8000, 0x561397680ef5, 0xc000ae9c80, 0xc0006f77b0)\n\t/usr/lib/golang/src/io/io.go:423 +0x12c\nio.Copy(...)\n\t/usr/lib/golang/src/io/io.go:382\nos/exec.(*Cmd).writerDescriptor.func1(0xc000ae9c80, 0xc0006f77b0)\n\t/usr/lib/golang/src/os/exec/exec.go:311 +0x65\nos/exec.(*Cmd).Start.func1(0xc00034f760, 0xc0009c6580)\n\t/usr/lib/golang/src/os/exec/exec.go:441 +0x29\ncreated by os/exec.(*Cmd).Start\n\t/usr/lib/golang/src/os/exec/exec.go:440 +0x630\n\ngoroutine 292 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d7408, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0007d2c98, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0007d2c80, 0xc0005c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0007d2c80, 0xc0005c1000, 0x1000, 0x1000, 0x0, 0x7f57200d7410, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0006e8690, 0xc0005c1000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc0004f86f0, 0xc0005c1000, 0x1000, 0x1000, 0xc033822c3360472c, 0x281e46092, 0x561399c14d00)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).fill(0xc000c0d920)\n\t/usr/lib/golang/src/bufio/bufio.go:101 +0x108\nbufio.(*Reader).Peek(0xc000c0d920, 0x4, 0x281e46092, 0x561399c14d00, 0x0, 0x0, 0x561399c14d00)\n\t/usr/lib/golang/src/bufio/bufio.go:139 +0x51\nnet/http.(*conn).serve(0xc000760960, 0x56139908cf08, 0xc000b6abc0)\n\t/usr/lib/golang/src/net/http/server.go:1977 +0xa47\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 106 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d6e98, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000c0d1b8, 0x72, 0x201, 0x200, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000c0d1a0, 0xc00069be00, 0x200, 0x200, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nos.(*File).read(...)\n\t/usr/lib/golang/src/os/file_posix.go:31\nos.(*File).Read(0xc0000108d8, 0xc00069be00, 0x200, 0x200, 0x2, 0x743a00006944676e, 0x0)\n\t/usr/lib/golang/src/os/file.go:117 +0x85\nbytes.(*Buffer).ReadFrom(0xc000a98480, 0x5613990593a8, 0xc0000108d8, 0x7f5720083028, 0xc000a98480, 0xc0006f7f01)\n\t/usr/lib/golang/src/bytes/buffer.go:204 +0xbe\nio.copyBuffer(0x561399056428, 0xc000a98480, 0x5613990593a8, 0xc0000108d8, 0x0, 0x0, 0x0, 0xc0006f7fa8, 0xc0006f7fa0, 0x0)\n\t/usr/lib/golang/src/io/io.go:409 +0x357\nio.Copy(...)\n\t/usr/lib/golang/src/io/io.go:382\nos/exec.(*Cmd).writerDescriptor.func1(0x561399030788, 0xc00010ee40)\n\t/usr/lib/golang/src/os/exec/exec.go:311 +0x65\nos/exec.(*Cmd).Start.func1(0xc00034f760, 0xc0009c6560)\n\t/usr/lib/golang/src/os/exec/exec.go:441 +0x29\ncreated by os/exec.(*Cmd).Start\n\t/usr/lib/golang/src/os/exec/exec.go:440 +0x630\n\ngoroutine 209 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d6be0, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc000c51278, 0x72, 0x8001, 0x8000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc000c51260, 0xc000cc8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nos.(*File).read(...)\n\t/usr/lib/golang/src/os/file_posix.go:31\nos.(*File).Read(0xc000132440, 0xc000cc8000, 0x8000, 0x8000, 0x88, 0x0, 0x0)\n\t/usr/lib/golang/src/os/file.go:117 +0x85\nio.copyBuffer(0x561399058e28, 0xc000b6e0d8, 0x5613990593a8, 0xc000132440, 0xc000cc8000, 0x8000, 0x8000, 0xc0007ed7a8, 0xc0007ed7a0, 0x0)\n\t/usr/lib/golang/src/io/io.go:423 +0x12c\nio.Copy(...)\n\t/usr/lib/golang/src/io/io.go:382\nos/exec.(*Cmd).writerDescriptor.func1(0x561399030788, 0xc00010f1a0)\n\t/usr/lib/golang/src/os/exec/exec.go:311 +0x65\nos/exec.(*Cmd).Start.func1(0xc00072c9a0, 0xc000c58620)\n\t/usr/lib/golang/src/os/exec/exec.go:441 +0x29\ncreated by os/exec.(*Cmd).Start\n\t/usr/lib/golang/src/os/exec/exec.go:440 +0x630\n\ngoroutine 597 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d7890, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0008a3018, 0x72, 0x0, 0x1, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0008a3000, 0xc000932491, 0x1, 0x1, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0008a3000, 0xc000932491, 0x1, 0x1, 0xc000315c20, 0x0, 0x0)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc000010aa8, 0xc000932491, 0x1, 0x1, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).backgroundRead(0xc000932480)\n\t/usr/lib/golang/src/net/http/server.go:692 +0x5a\ncreated by net/http.(*connReader).startBackgroundRead\n\t/usr/lib/golang/src/net/http/server.go:688 +0xd5\n\ngoroutine 476 [syscall, locked to thread]:\nsyscall.Syscall(0x4b, 0x27, 0x0, 0x0, 0x561397f2e27d, 0xc000c2c2a8, 0xc000b22000)\n\t/usr/lib/golang/src/syscall/asm_linux_amd64.s:20 +0x5\nsyscall.Fdatasync(0x27, 0x1000, 0x0)\n\t/usr/lib/golang/src/syscall/zsyscall_linux_amd64.go:393 +0x45\ngithub.com/containers/podman/vendor/go.etcd.io/bbolt.fdatasync(0xc000e7ee00, 0x1000, 0x1000)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/bolt_linux.go:9 +0x46\ngithub.com/containers/podman/vendor/go.etcd.io/bbolt.(*Tx).write(0xc000e781c0, 0x2e3a4874, 0x561399c14d00)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/tx.go:559 +0x47b\ngithub.com/containers/podman/vendor/go.etcd.io/bbolt.(*Tx).Commit(0xc000e781c0, 0x0, 0x0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/tx.go:185 +0x208\ngithub.com/containers/podman/vendor/go.etcd.io/bbolt.(*DB).Update(0xc000e7ee00, 0xc000db5230, 0x0, 0x0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/go.etcd.io/bbolt/db.go:701 +0x105\ngithub.com/containers/podman/libpod.(*BoltState).SaveContainer(0xc0006ce1e0, 0xc0007a
Jul 13 15:12:06 adrien-xps13 podman[310658]: 83c0, 0x0, 0x0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/boltdb_state.go:816 +0x2c5\ngithub.com/containers/podman/libpod.(*Container).save(0xc0007a83c0, 0x0, 0x0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/container_internal.go:786 +0x4c\ngithub.com/containers/podman/libpod.(*Container).NetworkDisconnect(0xc0007a83c0, 0xc000aa20c0, 0x40, 0xc000438130, 0xe, 0x0, 0x0, 0x0)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/networking_linux.go:1118 +0x649\ngithub.com/containers/podman/libpod.(*Runtime).DisconnectContainerFromNetwork(0xc0001e2fc0, 0xc000aa20c0, 0x40, 0xc000aa2015, 0xe, 0xc000c20000, 0xb200000000000000, 0xc000c11620)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/networking_linux.go:1219 +0xb0\ngithub.com/containers/podman/pkg/api/handlers/compat.Disconnect(0x561399082710, 0xc000da0000, 0xc0009e8a00)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/handlers/compat/networks.go:380 +0x1a5\ngithub.com/containers/podman/pkg/api/server.(*APIServer).APIHandler.func1.2(0x561399082710, 0xc000da0000, 0xc0009e8900)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/handler_api.go:66 +0x1004\ngithub.com/containers/podman/pkg/api/server.(*APIServer).APIHandler.func1(0x561399082710, 0xc000da0000, 0xc0009e8900)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/pkg/api/server/handler_api.go:69 +0xa8\nnet/http.HandlerFunc.ServeHTTP(0xc00000fe78, 0x561399082710, 0xc000da0000, 0xc0009e8900)\n\t/usr/lib/golang/src/net/http/server.go:2069 +0x46\ngithub.com/containers/podman/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc00049e000, 0x561399082710, 0xc000da0000, 0xc0009e8000)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/vendor/github.com/gorilla/mux/mux.go:210 +0xd3\nnet/http.serverHandler.ServeHTTP(0xc00033a900, 0x561399082710, 0xc000da0000, 0xc0009e8000)\n\t/usr/lib/golang/src/net/http/server.go:2887 +0xa5\nnet/http.(*conn).serve(0xc0007a8320, 0x56139908cf08, 0xc000a3a000)\n\t/usr/lib/golang/src/net/http/server.go:1952 +0x8cd\ncreated by net/http.(*Server).Serve\n\t/usr/lib/golang/src/net/http/server.go:3013 +0x3ad\n\ngoroutine 477 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d6f80, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0007dc298, 0x72, 0x0, 0x1, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0007dc280, 0xc0004905b1, 0x1, 0x1, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0007dc280, 0xc0004905b1, 0x1, 0x1, 0xc0005f0900, 0xc000c02798, 0x1)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc000010010, 0xc0004905b1, 0x1, 0x1, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).backgroundRead(0xc0004905a0)\n\t/usr/lib/golang/src/net/http/server.go:692 +0x5a\ncreated by net/http.(*connReader).startBackgroundRead\n\t/usr/lib/golang/src/net/http/server.go:688 +0xd5\n\ngoroutine 486 [IO wait]:\ninternal/poll.runtime_pollWait(0x7f57200d77a8, 0x72, 0xffffffffffffffff)\n\t/usr/lib/golang/src/runtime/netpoll.go:222 +0x65\ninternal/poll.(*pollDesc).wait(0xc0003fdd98, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:87 +0x47\ninternal/poll.(*pollDesc).waitRead(...)\n\t/usr/lib/golang/src/internal/poll/fd_poll_runtime.go:92\ninternal/poll.(*FD).Read(0xc0003fdd80, 0xc0005c8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/internal/poll/fd_unix.go:166 +0x1d8\nnet.(*netFD).Read(0xc0003fdd80, 0xc0005c8000, 0x8000, 0x8000, 0x561399c4c548, 0xc000e0eda8, 0x561397688f2a)\n\t/usr/lib/golang/src/net/fd_posix.go:55 +0x51\nnet.(*conn).Read(0xc0006e8ea8, 0xc0005c8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/net/net.go:183 +0x93\nnet/http.(*connReader).Read(0xc0007d9e60, 0xc0005c8000, 0x8000, 0x8000, 0x8000, 0x561398d87660, 0x1)\n\t/usr/lib/golang/src/net/http/server.go:800 +0x1bf\nbufio.(*Reader).Read(0xc000315500, 0xc0005c8000, 0x8000, 0x8000, 0x0, 0x0, 0x0)\n\t/usr/lib/golang/src/bufio/bufio.go:213 +0x144\ngithub.com/containers/podman/utils.CopyDetachable(0x561399059008, 0xc000132050, 0x561399056328, 0xc000b2e4f0, 0xc0000fc258, 0x2, 0x8, 0xc000967000, 0x1000, 0x1000)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/utils/utils.go:61 +0xa8\ngithub.com/containers/podman/libpod.(*ConmonOCIRuntime).HTTPAttach.func6(0xc000132050, 0xc000b2e4f0, 0xc0000fc258, 0x2, 0x8, 0xc0004e4180)\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:690 +0x88\ncreated by github.com/containers/podman/libpod.(*ConmonOCIRuntime).HTTPAttach\n\t/builddir/build/BUILD/podman-3.2.2/_build/src/github.com/containers/podman/libpod/oci_conmon_linux.go:689 +0x7f7\n"

@baude
Copy link
Member

baude commented Jul 13, 2021

ish! thanks

@djnotes
Copy link
Author

djnotes commented Jul 14, 2021

Adding to my previous comment, the freezing happens only with rootless Podman. Preceding the commands with sudo fixes the problem. For example, sudo podman version, sudo podman --debug, and sudo docker-compose up

@djnotes
Copy link
Author

djnotes commented Jul 14, 2021

Despite the aforementioned, Podman Compose still works with Podman.

@Luap99 Luap99 self-assigned this Jul 14, 2021
@Luap99
Copy link
Member

Luap99 commented Jul 14, 2021

I know how to fix this. I will work on it tomorrow.

@djnotes
Copy link
Author

djnotes commented Jul 14, 2021

Seems like restarting user-level podman socket remediates this.

@Luap99 Luap99 added the In Progress This issue is actively being worked by the assignee, please do not work on this at this time. label Jul 15, 2021
Luap99 added a commit to Luap99/libpod that referenced this issue Jul 15, 2021
There was an race condition when calling `GetRootlessCNINetNs()`. It
created the rootless cni directory before it got locked. Therefore
another process could have called cleanup and removed this directory
before it was used resulting in errors. The lockfile got moved into the
XDG_RUNTIME_DIR directory to prevent a panic when the parent dir was
removed by cleanup.

Fixes containers#10930
Fixes containers#10922

To make this even more robust `GetRootlessCNINetNs()` will now return
locked. This guarantees that we can run `Do()` after `GetRootlessCNINetNs()`
before another process could have called `Cleanup()` in between.

[NO TESTS NEEDED] CI is flaking, hopefully this will fix it.

Signed-off-by: Paul Holzinger <[email protected]>
Luap99 added a commit to Luap99/libpod that referenced this issue Jul 15, 2021
There was an race condition when calling `GetRootlessCNINetNs()`. It
created the rootless cni directory before it got locked. Therefore
another process could have called cleanup and removed this directory
before it was used resulting in errors. The lockfile got moved into the
XDG_RUNTIME_DIR directory to prevent a panic when the parent dir was
removed by cleanup.

Fixes containers#10930
Fixes containers#10922

To make this even more robust `GetRootlessCNINetNs()` will now return
locked. This guarantees that we can run `Do()` after `GetRootlessCNINetNs()`
before another process could have called `Cleanup()` in between.

[NO TESTS NEEDED] CI is flaking, hopefully this will fix it.

Signed-off-by: Paul Holzinger <[email protected]>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 21, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 21, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
In Progress This issue is actively being worked by the assignee, please do not work on this at this time. kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants