podman-exec tty resize: stty: standard input

[edsantiago]

Fallout from #10683 (tty resize). The test even flaked in that very same PR:

# # podman exec -it mystty stty size
# stty: standard input

# #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
# #|     FAIL: stty under podman exec reads the correct dimensions
# #| expected: '41 62'
# #|   actual: 'stty: standard input
'
# #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

sys: podman detects correct tty size

• ubuntu-2104 : sys podman ubuntu-2104 root host
  • PR Fix resize race with podman exec -it #10683
    • 06-16 11:55

...and it just triggered in a new PR, also ubuntu-2104.

[edsantiago]

Yikes - the #10688 flake was a triple:

• 06-17 12:26
• 06-17 11:20
• 06-17 07:46

[Luap99]

@edsantiago Are you able to reproduce locally?

[edsantiago]

I have not been able to reproduce, because I'm plagued by #10701 on my laptop.

[mheon]

I'm struggling to see how this could happen - the resize is now happening in a way that can't race.

[edsantiago]

I just reproduced on f34, podman-3.2.1-1.fc34 rootless. Passed on retry.

[edsantiago]

Reproduced on same f34 system as above, this time in podman-remote rootless. Does the warning message help?

   $ podman-remote run -it --name mystty quay.io/libpod/testimage:20210427 stty size
   stty: standard input
   time="2021-06-17T16:38:39-04:00" level=warning msg="failed to resize TTY: can only resize created or running containers: container state improper"
   #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
   #|     FAIL: stty under podman reads the correct dimensions
   #| expected: '55 28'
'  #|   actual: 'stty: standard input
   #|         > 'time="2021-06-17T16:38:39-04:00" level=warning msg="failed to resize TTY: can only resize created or running containers: container state improper"'
   #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[mheon]

OK, so the resize is happening too soon, presumably? That at least makes sense.

[edsantiago]

Recommendation: try wait_for_ready.

[rhatdan]

@Luap99 PTAL

[Luap99]

Reproduced on same f34 system as above, this time in podman-remote rootless. Does the warning message help?

   $ podman-remote run -it --name mystty quay.io/libpod/testimage:20210427 stty size
   stty: standard input
   time="2021-06-17T16:38:39-04:00" level=warning msg="failed to resize TTY: can only resize created or running containers: container state improper"
   #/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
   #|     FAIL: stty under podman reads the correct dimensions
   #| expected: '55 28'
'  #|   actual: 'stty: standard input
   #|         > 'time="2021-06-17T16:38:39-04:00" level=warning msg="failed to resize TTY: can only resize created or running containers: container state improper"'
   #\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Well you managed to reproduced this issue with podman-remote run so this is not the same problem as podman exec.

@mheon I believe the race is between podman and conmon. The resize call will simply write to a named pipe and after that podman will write to another pipe to signal conmon that it should start the process. I can imagine a case where conmon reads the start pipe before the resize pipe.

Conmon never signals podman if the resize even worked.

[mheon]

Could we also be looking at an internal latency in Conmon, where the process is not immediately ready after the start pipe is read?

Regardless, this sounds like a reasonable conclusion. I don't see an easy way around it, though - Conmon doesn't tell us that the start signal was received, either, so there's no easy way to syncronize

[vrothberg]

Any progress on that? It's a quite consistent and stubborn flake in the main branch at the moment.

[vrothberg]

I opened #10758 to disable the flaky tests. We need to make sure to reenable the tests once the underlying issue is fixed (I made sure to drop a comment).

I usually feel strongly about disabling tests but the flakes are just too frequent.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@edsantiago @vrothberg Any update on this issue?

[edsantiago]

The multiarch testing group is still seeing this. Podman 3.2.3 on s390x; I don't know if it's Fedora or RHEL. If I read the log correctly, August 19.

[vrothberg]

Let's wait for the results on v3.3.0. We increased the sizes of signal buffers "in hope" to resolve such issues which made it into v3.3.0.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Well it is a month later, so I am going to assume this is fixed. Reopen if it happens again.

[edsantiago]

I've got a 1minutetip VM right now that's reproducing it super-easily. Ping me for access.

The "remote" connection (in comments above) was me grasping at straws. I don't know if this merits a new issue. My gut tells me no, that the common factor is "stty is broken", not whether it's via exec or remote.

[Luap99]

My comment is still valid.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

I can no longer reproduce this: not with the podman run reproducer above (Dec 7), and not with exec either. Not on f35 with compiled podman, nor on Rawhide with 4.0rc6.

Is it time to reenable the commented-out exec ... stty test, and see what happens?

[edsantiago]

Flaked in the very PR that was going to reintroduce the test. Problem still exists, even if I can't reproduce it on my laptop or a Rawhide VM.

[Luap99]

I guess this can be fixed with conmon-rs in the future.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[edsantiago]

After a long absence, this is back. rawhide rootless:

not ok 416 podman detects correct tty size
...
$ podman run -it --name mystty quay.io/libpod/testimage:20221018 stty size
stty: standard input

#/vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv
#|     FAIL: stty under podman run reads the correct dimensions
#| expected: '31 32
'
#|   actual: 'stty: standard input
'
#\^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[Luap99]

Yeah at this point I don't think this is ever going to be fixed, we would need a bidirectional channel to get a success response back from conmon and only then start the container. I guess conmon-rs is your best hope here.