podman exec may truncate stdout/stderr #9096

liewegas · 2021-01-26T00:02:40Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

BUG REPORT

Description

podman run and exec will truncate stdout. Sometimes.

This seems to reproduce with the ubuntu build of podman 2.2.1 available at https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04/

Steps to reproduce the issue:

podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test

root@teuthology:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test 
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.753493 s, 664 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 00:01 /tmp/test
root@teuthology:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test 
-rw-r--r-- 1 root root 1601 Jan 26 00:01 /tmp/test
root@teuthology:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test 
-rw-r--r-- 1 root root 2518 Jan 26 00:01 /tmp/test
root@teuthology:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test 
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.794628 s, 629 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 00:01 /tmp/test
root@teuthology:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test 
-rw-r--r-- 1 root root 1836 Jan 26 00:01 /tmp/test

root@teuthology:~# podman -v
podman version 2.2.1
root@teuthology:~# grep PRETTY /etc/os-release 
PRETTY_NAME="Ubuntu 18.04.4 LTS"

Note that we haven't observed the same problem on fedora or centos--only the ubuntu 18.04 and 20.04 hosts using the builds from opensuse.org.

Describe the results you received:

/tmp/test may not be 500000 bytes. For example, on my ubuntu 18.04 and 20.04 hosts,

Describe the results you expected:

/tmp/test should be 500000 bytes, every time.

Additional information you deem important (e.g. issue happens only occasionally):

We see the same behavior on both 18.04 and 20.04 (using the opensuse.org builds). We do not see any problems on the fedora and centos machines with the same version of podman.

Output of podman version:

podman version 2.2.1

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.24, commit: '
  cpus: 16
  distribution:
    distribution: ubuntu
    version: "18.04"
  eventLogger: journald
  hostname: teuthology
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 4.15.0-128-generic
  linkmode: dynamic
  memFree: 9252192256
  memTotal: 37953269760
  ociRuntime:
    name: runc
    package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc'
    path: /usr/lib/cri-o-runc/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  rootless: false
  slirp4netns:
    executable: ""
    package: ""
    version: ""
  swapFree: 28755144704
  swapTotal: 34359734272
  uptime: 894h 12m 8.1s (Approximately 37.25 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev
  graphRoot: /var/lib/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /var/run/containers/storage
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 2.2.1~4 amd64 [installed]

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

bare metal servers

The text was updated successfully, but these errors were encountered:

mheon · 2021-01-26T05:22:23Z

This usually tends to be Conmon, but that’s a quite recent Conmon build which makes that a bit doubtful. Does this not occur with the run or attach commands? Both of those should use the same code paths.

…

On Mon, Jan 25, 2021 at 19:03 Sage Weil ***@***.***> wrote: *Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)* BUG REPORT *Description* podman exec will truncate stdout. Sometimes. This seems to reproduce with the ubuntu build of podman 2.2.1 available at https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04/ *Steps to reproduce the issue:* 1. podman run --rm --entrypoint /bin/bash quay.ceph.io/ceph-ci/ceph:d8ced8b53927915a604da2faa03905eff41f1774 -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test ***@***.***:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test 500000+0 records in 500000+0 records out 500000 bytes (500 kB, 488 KiB) copied, 0.753493 s, 664 kB/s -rw-r--r-- 1 root root 500000 Jan 26 00:01 /tmp/test ***@***.***:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test -rw-r--r-- 1 root root 1601 Jan 26 00:01 /tmp/test ***@***.***:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test -rw-r--r-- 1 root root 2518 Jan 26 00:01 /tmp/test ***@***.***:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test 500000+0 records in 500000+0 records out 500000 bytes (500 kB, 488 KiB) copied, 0.794628 s, 629 kB/s -rw-r--r-- 1 root root 500000 Jan 26 00:01 /tmp/test ***@***.***:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test -rw-r--r-- 1 root root 1836 Jan 26 00:01 /tmp/test ***@***.***:~# podman -v podman version 2.2.1 ***@***.***:~# grep PRETTY /etc/os-release PRETTY_NAME="Ubuntu 18.04.4 LTS" Note that we haven't observed the same problem on fedora or centos--only the ubuntu 18.04 and 20.04 hosts using the builds from opensuse.org. *Describe the results you received:* /tmp/test may not be 500000 bytes. For example, on my ubuntu 18.04 and 20.04 hosts, *Describe the results you expected:* /tmp/test should be 500000 bytes, every time. *Additional information you deem important (e.g. issue happens only occasionally):* We see the same behavior on both 18.04 and 20.04 (using the opensuse.org builds). We do not see any problems on the fedora and centos machines with the same version of podman. *Output of podman version:* podman version 2.2.1 *Output of podman info --debug:* host: arch: amd64 buildahVersion: 1.18.0 cgroupManager: systemd cgroupVersion: v1 conmon: package: 'conmon: /usr/libexec/podman/conmon' path: /usr/libexec/podman/conmon version: 'conmon version 2.0.24, commit: ' cpus: 16 distribution: distribution: ubuntu version: "18.04" eventLogger: journald hostname: teuthology idMappings: gidmap: null uidmap: null kernel: 4.15.0-128-generic linkmode: dynamic memFree: 9252192256 memTotal: 37953269760 ociRuntime: name: runc package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc' path: /usr/lib/cri-o-runc/sbin/runc version: 'runc version spec: 1.0.1-dev' os: linux remoteSocket: path: /run/podman/podman.sock rootless: false slirp4netns: executable: "" package: "" version: "" swapFree: 28755144704 swapTotal: 34359734272 uptime: 894h 12m 8.1s (Approximately 37.25 days) registries: search: - docker.io - quay.io store: configFile: /etc/containers/storage.conf containerStore: number: 2 paused: 0 running: 2 stopped: 0 graphDriverName: overlay graphOptions: overlay.mountopt: nodev graphRoot: /var/lib/containers/storage graphStatus: Backing Filesystem: xfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" imageStore: number: 3 runRoot: /var/run/containers/storage volumePath: /var/lib/containers/storage/volumes version: APIVersion: 2.1.0 Built: 0 BuiltTime: Thu Jan 1 00:00:00 1970 GitCommit: "" GoVersion: go1.15.2 OsArch: linux/amd64 Version: 2.2.1 *Package info (e.g. output of rpm -q podman or apt list podman):* podman/unknown,now 2.2.1~4 amd64 [installed] *Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?* Yes *Additional environment details (AWS, VirtualBox, physical, etc.):* bare metal servers — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9096>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB3AOCAGZKBESUTSSK5YWCDS3YBDDANCNFSM4WSR7Y2A> .

vrothberg · 2021-01-26T09:14:22Z

@giuseppe could this be runc?

vrothberg · 2021-01-26T09:14:30Z

@lsm5 FYI

liewegas · 2021-01-26T15:18:22Z

This usually tends to be Conmon, but that’s a quite recent Conmon build which makes that a bit doubtful. Does this not occur with the run or attach commands? Both of those should use the same code paths.
…
On Mon, Jan 25, 2021 at 19:03 Sage Weil @.> wrote: Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line) BUG REPORT Description podman exec will truncate stdout. Sometimes. This seems to reproduce with the ubuntu build of podman 2.2.1 available at https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_18.04/ Steps to reproduce the issue: 1. podman run --rm --entrypoint /bin/bash quay.ceph.io/ceph-ci/ceph:d8ced8b53927915a604da2faa03905eff41f1774 -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test @.:# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test 500000+0 records in 500000+0 records out 500000 bytes (500 kB, 488 KiB) copied, 0.753493 s, 664 kB/s -rw-r--r-- 1 root root 500000 Jan 26 00:01 /tmp/test @.***:# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test -rw-r--r-- 1 root root 1601 Jan 26 00:01 /tmp/test @.:~# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test -rw-r--r-- 1 root root 2518 Jan 26 00:01 /tmp/test @.:# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test 500000+0 records in 500000+0 records out 500000 bytes (500 kB, 488 KiB) copied, 0.794628 s, 629 kB/s -rw-r--r-- 1 root root 500000 Jan 26 00:01 /tmp/test @.***:# podman run --rm --entrypoint /bin/bash ubuntu -c 'dd if=/dev/zero count=500000 bs=1' > /tmp/test ; ls -al /tmp/test -rw-r--r-- 1 root root 1836 Jan 26 00:01 /tmp/test @.:~# podman -v podman version 2.2.1 @.:# grep PRETTY /etc/os-release PRETTY_NAME="Ubuntu 18.04.4 LTS" Note that we haven't observed the same problem on fedora or centos--only the ubuntu 18.04 and 20.04 hosts using the builds from opensuse.org. Describe the results you received: /tmp/test may not be 500000 bytes. For example, on my ubuntu 18.04 and 20.04 hosts, Describe the results you expected: /tmp/test should be 500000 bytes, every time. Additional information you deem important (e.g. issue happens only occasionally): We see the same behavior on both 18.04 and 20.04 (using the opensuse.org builds). We do not see any problems on the fedora and centos machines with the same version of podman. Output of podman version: podman version 2.2.1 Output of podman info --debug: host: arch: amd64 buildahVersion: 1.18.0 cgroupManager: systemd cgroupVersion: v1 conmon: package: 'conmon: /usr/libexec/podman/conmon' path: /usr/libexec/podman/conmon version: 'conmon version 2.0.24, commit: ' cpus: 16 distribution: distribution: ubuntu version: "18.04" eventLogger: journald hostname: teuthology idMappings: gidmap: null uidmap: null kernel: 4.15.0-128-generic linkmode: dynamic memFree: 9252192256 memTotal: 37953269760 ociRuntime: name: runc package: 'cri-o-runc: /usr/lib/cri-o-runc/sbin/runc' path: /usr/lib/cri-o-runc/sbin/runc version: 'runc version spec: 1.0.1-dev' os: linux remoteSocket: path: /run/podman/podman.sock rootless: false slirp4netns: executable: "" package: "" version: "" swapFree: 28755144704 swapTotal: 34359734272 uptime: 894h 12m 8.1s (Approximately 37.25 days) registries: search: - docker.io - quay.io store: configFile: /etc/containers/storage.conf containerStore: number: 2 paused: 0 running: 2 stopped: 0 graphDriverName: overlay graphOptions: overlay.mountopt: nodev graphRoot: /var/lib/containers/storage graphStatus: Backing Filesystem: xfs Native Overlay Diff: "true" Supports d_type: "true" Using metacopy: "false" imageStore: number: 3 runRoot: /var/run/containers/storage volumePath: /var/lib/containers/storage/volumes version: APIVersion: 2.1.0 Built: 0 BuiltTime: Thu Jan 1 00:00:00 1970 GitCommit: "" GoVersion: go1.15.2 OsArch: linux/amd64 Version: 2.2.1 Package info (e.g. output of rpm -q podman or apt list podman): podman/unknown,now 2.2.14 amd64 [installed] Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? Yes Additional environment details (AWS, VirtualBox, physical, etc.): bare metal servers — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#9096>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB3AOCAGZKBESUTSSK5YWCDS3YBDDANCNFSM4WSR7Y2A .

Yeah, podman exec also seems affected, failing about 20% of the time (though I suspect this is highly dependent on the system, how loady it is, etc.):

root@teuthology:~# while true; do podman exec -i 5fef984f3594 dd if=/dev/zero bs=1 count=500000 > /tmp/test ; ls -al /tmp/test ; done
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.687549 s, 727 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 15:16 /tmp/test
-rw-r--r-- 1 root root 132692 Jan 26 15:16 /tmp/test
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.680406 s, 735 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 15:16 /tmp/test
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.684233 s, 731 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 15:16 /tmp/test
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.716087 s, 698 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 15:16 /tmp/test
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.680967 s, 734 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 15:16 /tmp/test
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.704278 s, 710 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 15:16 /tmp/test
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.692762 s, 722 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 15:16 /tmp/test
-rw-r--r-- 1 root root 19701 Jan 26 15:16 /tmp/test
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.703558 s, 711 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 15:16 /tmp/test
500000+0 records in
500000+0 records out
500000 bytes (500 kB, 488 KiB) copied, 0.681633 s, 734 kB/s
-rw-r--r-- 1 root root 500000 Jan 26 15:16 /tmp/test

sebastian-philipp · 2021-02-01T12:51:32Z

https://tracker.ceph.com/issues/49076

vrothberg · 2021-02-01T12:54:37Z

@containers/podman-maintainers FYI

edsantiago · 2021-02-01T13:00:31Z

~~There are gating tests for this condition. Unfortunately they're disabled for podman-remote due to #7360 which remains open:~~

podman/test/system/075-exec.bats

Lines 61 to 62 in b045c17

    
           @test "podman exec - cat from stdin" { 
        
               skip_if_remote "FIXME: pending #7360"

EDIT: my mistake, I misread this issue as related to podman-remote

mheon · 2021-02-01T15:31:42Z

@edsantiago Is the issue actually with podman-remote here? This seems to be local Podman - which should be much more stable, since there's no server-side traffic forwarding going on.

edsantiago · 2021-02-01T15:39:00Z

@mheon my mistake again - I seem to be having reading comprehension problems today! I've edited my comment. Sorry for the confusion.

edsantiago · 2021-02-02T21:16:40Z

Is this one also closed by containers/conmon#237 ?

mheon · 2021-02-02T21:21:45Z

Yes.

This actually tests conmon, not podman; but that's the whole point of system tests in the first place: if a problem exists, we want to fail loudly, no matter whose fault it is. (I can't get this to fail on my f33 laptop; OP on containers#9096 claims it only fails on Ubuntu. We'll see what happens in CI). Signed-off-by: Ed Santiago <[email protected]>

System test for #9096 (truncated stdout)

liewegas · 2021-02-08T22:28:28Z

Thank you, everyone!

"conmon" is a binary started withing podman containers. The version provided with bullseye has a serious issue sometimes causing the output of "podman [run|exec]" to be truncated. containers/podman#9096 We replace this binary with a more up-to-date version statically compiled using the nix method. https://github.com/containers/conmon#nix (note: nix compilation is very long.)

I think that on 22.04, we are hitting containers/podman#9096

liewegas changed the title ~~podman exec may truncate stdout~~ podman exec may truncate stdout/stderr Jan 26, 2021

liewegas mentioned this issue Jan 26, 2021

qa/suites/rados/cephadm: drop ubuntu + podman from test matrix for now ceph/ceph#39087

Merged

mheon mentioned this issue Feb 1, 2021

podman hangs on "receiving" too much output to console #9183

Closed

edsantiago mentioned this issue Feb 2, 2021

System test for #9096 (truncated stdout) #9208

Merged

mheon closed this as completed Feb 2, 2021

openshift-merge-robot added a commit that referenced this issue Feb 3, 2021

Merge pull request #9208 from edsantiago/bats

4945abb

System test for #9096 (truncated stdout)

sebastian-philipp mentioned this issue Mar 18, 2021

mgr/cephadm: retry after JSONDecodeError in wait_for_mgr_restart() ceph/ceph#40203

Merged

3 tasks

ondrejbudai added a commit to ondrejbudai/github-actions-osbuild-composer-demo that referenced this issue Jun 6, 2023

.github/workflows/build.yml: roll back to ubuntu 20.04

98063b1

I think that on 22.04, we are hitting containers/podman#9096

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman exec may truncate stdout/stderr #9096

podman exec may truncate stdout/stderr #9096

liewegas commented Jan 26, 2021 •

edited

Loading

mheon commented Jan 26, 2021 via email

vrothberg commented Jan 26, 2021

vrothberg commented Jan 26, 2021

liewegas commented Jan 26, 2021

sebastian-philipp commented Feb 1, 2021

vrothberg commented Feb 1, 2021

edsantiago commented Feb 1, 2021 •

edited

Loading

mheon commented Feb 1, 2021

edsantiago commented Feb 1, 2021

edsantiago commented Feb 2, 2021

mheon commented Feb 2, 2021

liewegas commented Feb 8, 2021

podman exec may truncate stdout/stderr #9096

podman exec may truncate stdout/stderr #9096

Comments

liewegas commented Jan 26, 2021 • edited Loading

mheon commented Jan 26, 2021 via email

vrothberg commented Jan 26, 2021

vrothberg commented Jan 26, 2021

liewegas commented Jan 26, 2021

sebastian-philipp commented Feb 1, 2021

vrothberg commented Feb 1, 2021

edsantiago commented Feb 1, 2021 • edited Loading

mheon commented Feb 1, 2021

edsantiago commented Feb 1, 2021

edsantiago commented Feb 2, 2021

mheon commented Feb 2, 2021

liewegas commented Feb 8, 2021

liewegas commented Jan 26, 2021 •

edited

Loading

edsantiago commented Feb 1, 2021 •

edited

Loading