A warning message can appear when using podman exec command

[Lordryte]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

A new WARNING message appeared recently, since last update to podman V3 when using the podman exec command.

Steps to reproduce the issue:

1. podman run -dit --name bugtest ubuntu

2. podman exec -lt echo "toto"

Describe the results you received:

toto
WARN[0000] Error resizing exec session 68dcfedf1a47fd34ae83b7460bf7a1f3d291fecb92d7f16c9fcb6b26e4704bb4: could not open ctl file for terminal resize for container f0284de404836b13c027a96e42324fcd98e6e7bfe8f89e4c02ca928b65224ad4: open /home/podman/.local/share/containers/storage/overlay-containers/f0284de404836b13c027a96e42324fcd98e6e7bfe8f89e4c02ca928b65224ad4/userdata/68dcfedf1a47fd34ae83b7460bf7a1f3d291fecb92d7f16c9fcb6b26e4704bb4/ctl: no such device or address

Describe the results you expected:
toto

Additional information you deem important (e.g. issue happens only occasionally):

The warning message appearance is intermitent, sometimes no warning message is displayed.
The usecase is rootless podman.

Output of podman version:

Version:      3.0.0
API Version:  3.0.0
Go Version:   go1.15.2
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.19.2
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.26, commit: '
  cpus: 10
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: journald
  hostname: 
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1002
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.4.0-65-generic
  linkmode: dynamic
  memFree: 37038759936
  memTotal: 63221137408
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version 0.17.7-5502-dirty
      commit: fd582c529489c0738e7039cbc036781d1d039014
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    selinuxEnabled: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 1h 44m 30.62s (Approximately 0.04 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/podman/.config/containers/storage.conf
  containerStore:
    number: 23
    paused: 0
    running: 22
    stopped: 1
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 1.4
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/podman/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 87
  runRoot: /run/user/1001/containers
  volumePath: /home/podman/.local/share/containers/storage/volumes
version:
  APIVersion: 3.0.0
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 3.0.0

Package info (e.g. output of rpm -q podman or apt list podman):

podman/unknown,now 100:3.0.0-4 amd64 [installed]
podman/unknown 100:3.0.0-4 arm64
podman/unknown 100:3.0.0-4 armhf
podman/unknown 100:3.0.0-4 s390x

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

VPS

[Lordryte]

FYI The warning messages are still there in 3.0.1.

[vrothberg]

Thanks for reaching out, @Lordryte! I'll take a look what's going on.

[vrothberg]

@giuseppe, can you advice how to resolve the issue? Pinging you since you've been working in that parts of the code.

[mheon]

@vrothberg Best guess is this is a race - we are resizing before the exec session's conmon is running so the file is not there yet.

[giuseppe]

it is a race when the process is shortlived so it runs and dies before podman can connect to the terminal.

Try: podman exec -lt sh -c 'sleep 2; echo toto'

How can we make sure we attach before the exec starts? Should we ignore the error?

[StayPirate]

same issue on my side

> podman container exec -ti -w $(pwd) secbox echo test
test
WARN[0000] Error resizing exec session bad8229bc0297dea1a123f5db3a75ba72814a0a4674a3910e51371c6621ba402: could not open ctl file for terminal resize for container eeb4057902915c34a81722985390309d2e2aa3caaeb7d3363a3188ee7a880899: open /var/lib/containers/storage/overlay-containers/eeb4057902915c34a81722985390309d2e2aa3caaeb7d3363a3188ee7a880899/userdata/bad8229bc0297dea1a123f5db3a75ba72814a0a4674a3910e51371c6621ba402/ctl: no such device or address

[rhatdan]

Can we just check for this specific error, ENXIO, and drop to DEBUG?

[mheon]

No, this is a real bug - the container's terminal has an incorrect size (0 0) set if we get this.

[rhatdan]

But this is only happening, on short running exec commands, correct? Is there a way to tell if the exec command has completed by the time we are checking?

[chainmail]

I do not know if this is relevant, but it might be. We use a scripted solution to run containerised builds on local workstations for devs, etc. While implementing docker I ran across this bug report and found something which I do not think has been referenced.
We use a short lived command to check the messages content from the build scripts on Fedora 33 (all up to date):
podman version:
Version: 3.0.1 API Version: 3.0.0 Go Version: go1.15.8 Built: Fri Feb 19 17:56:17 2021 OS/Arch: linux/amd64

Command failing:
podman exec -it "mycontainer" /usr/bin/grep -c 'FAILED' /var/log/messages 2>/dev/null |wc -c
Assuming the search term is not matched, this should return 2, like "echo 0 |wc -c"
Actually it returns 3, and subsequent evaluations (assuming an integer result) fail because it is no longer a number.
Using this as a work around returns the correct result:
podman exec -it "mycontainer" /usr/bin/grep -c 'FAILED' /var/log/messages 2>/dev/null |dos2unix |wc -c

As I said, I do not know if this is relevant but I wondered if the extra character (I am assuming ^M) is related to the reported terminal size condition? If not, I'll raise another bug report.

Thanks for your efforts with podman. It really is appreciated.

[StayPirate]

This is not related to this bug, you can find the explanation to the behavior you mentioned and a better solution than using dos2unix here

[StayPirate]

No, this is a real bug - the container's terminal has an incorrect size (0 0) set if we get this.

Is there any update with this one? Or could you at least suggest a workaround to implement until a fix will be issued?

I tried podman container exec -ti container_name bash -c "resize &>/dev/null && $@" as suggested here #9780 (comment), but with no luck.

[mheon]

The original bug, with echo and other short-running commands, should be solved as of #9942 which should go into v3.1.1 later this week.

[mheon]

As such, I'm going to close.