Approval sometimes doesn't show up

[dylanratcliffe]

This seems to be a similar race condition to the ones we were seeing before. Occasionally when running overmind terraform plan the approval box doesn't show up and the command appears to hang, then other times it works. I have tried this a few times and the behaviour appears to be intermittent

Try 1: Worked

Try 2: Hung - teabug log attached

[dylanratcliffe]

teabug.log

[DavidS-ovm]

Interesting, cmd.askForApprovalMsg is received and then the process hangs. Will have to repro locally to debug deeper.

[DavidS-ovm]

I wasn't able to repro this in like 10 tries.

Can you exclude that this was a rendering issue (e.g. resizing the terminal at an inconvenient time)?

Can you confirm that the process was truly hanging (no reaction to inputs)? Specifically no reactions to (in this order): regular letters, pressing enter, pressing escape, pressing Ctrl+C.

If you have a process hanging like this the next time, please try to grab a goroutine dump by sending a SIQUIT or SIGABRT as described here: https://stackoverflow.com/a/35290196/4918

[dylanratcliffe]

Can you exclude that this was a rendering issue (e.g. resizing the terminal at an inconvenient time)?

I ran it again without resizing (I had never resized) and it's still happening

Can you confirm that the process was truly hanging (no reaction to inputs)? Specifically no reactions to (in this order):

Session 1:

• regular letters: nothing
• pressing enter: ends the command with exit 0

Session 2:

• pressing escape: nothing
• pressing Ctrl+C. prints the following:

┃ Do you want to perform these actions?                                         
┃ Terraform will perform the actions described above.                           
┃ Only 'yes' will be accepted to approve.                                       
┃ >                                                                             
                                                                                
enter submit

^ This was not visible before pressing Ctrl+C

If you have a process hanging like this the next time, please try to grab a goroutine dump by sending a SIQUIT or SIGABRT as described here: https://stackoverflow.com/a/35290196/4918

I can get a dump but it's ~7000 lines in this horiffic format:

                                                                /usr/local/go/src/runtime/chan.go:447 +0x14 fp=0x4000f53ef0 sp=0x4000f53ec0 pc=0x1aae4
                                                                                                                                                      github.com/sourcegraph/conc/pool.(*Pool).worker(0x400299f8d0?)
                                                /go/pkg/mod/github.com/sourcegraph/[email protected]/pool/pool.go:153 +0x74 fp=0x4000f53f40 sp=0x4000f53ef0 pc=0x2c19694
                                                                                                                                                                  github.com/sourcegraph/conc/pool.(*Pool).worker-fm()
                                                <autogenerated>:1 +0x28 fp=0x4000f53f60 sp=0x4000f53f40 pc=0x2c197c8
                                                                                                                    github.com/sourcegraph/conc/panics.(*Catcher).Try(0x4938260?, 0x40005a4e00?)
                                /go/pkg/mod/github.com/sourcegraph/[email protected]/panics/panics.go:23 +0x50 fp=0x4000f53f90 sp=0x4000f53f60 pc=0x2c18890
                                                                                                                                                     github.com/sourcegraph/conc.(*WaitGroup).Go.func1()
                                        /go/pkg/mod/github.com/sourcegraph/[email protected]/waitgroup.go:32 +0x58 fp=0x4000f53fd0 sp=0x4000f53f90 pc=0x2c19268
                                                                                                                                                         runtime.goexit({})
        /usr/local/go/src/runtime/asm_arm64.s:1222 +0x4 fp=0x4000f53fd0 sp=0x4000f53fd0 pc=0x84144
                                                                                                  created by github.com/sourcegraph/conc.(*WaitGroup).Go in goroutine 3460
        /go/pkg/mod/github.com/sourcegraph/[email protected]/waitgroup.go:30 +0x7c

                                                                            r0      0x6553740
                                                                                             r1      0x80
                                                                                                         r2      0x0
                                                                                                                    r3      0x0
                                                                                                                               r4      0x0
                                                                                                                                          r5      0x0
                                                                                                                                                     r6      0xc
                                                                                                                                                                r7      0x0
   r8      0x62
               r9      0x1
                          r10     0x0
                                     r11     0xa
                                                r12     0x0
                                                           r13     0x1
                                                                      r14     0x400210c750
                                                                                          r15     0x400210c7b0
                                                                                                              r16     0xffffe7d9f3a0
                                                                                                                                    r17     0xffffe859d3d0
                                                                                                                                                          r18     0xffffafc25670
        r19     0x400279a9a8
                            r20     0xffffe859d2d0
                                                  r21     0x6553600
                                                                   r22     0x1
                                                                              r23     0x4002797c60
                                                                                                  r24     0x4002795590
                                                                                                                      r25     0x400009bcc0
                                                                                                                                          r26     0x3ecc968
                                                                                                                                                           r27     0x63ab000
    r28     0x6551280
                     r29     0xffffe859d288
                                           lr      0x493ac
                                                          sp      0xffffe859d290
                                                                                pc      0x855bc
                                                                                               fault   0x0
                                                                                                          vscode ➜ /workspace/terraform-example (main) $ 

[DavidS-ovm]

Can you confirm that the process was truly hanging (no reaction to inputs)? Specifically no reactions to (in this order):

Session 1:

* **regular letters:** nothing

* **pressing enter:** ends the command with exit 0

Session 2:

* **pressing escape:** nothing

* **pressing Ctrl+C.** prints the following:

┃ Do you want to perform these actions?                                         
┃ Terraform will perform the actions described above.                           
┃ Only 'yes' will be accepted to approve.                                       
┃ >                                                                             
                                                                                
enter submit

^ This was not visible before pressing Ctrl+C

Since it does seem to respond to inputs, this is a rendering issue, not a "hang" per-se. A screenshot would be beneficial, and how the program reacts to resizing the terminal.

If you have a process hanging like this the next time, please try to grab a goroutine dump by sending a SIQUIT or SIGABRT as described here: https://stackoverflow.com/a/35290196/4918

I can get a dump but it's ~7000 lines in this horiffic format:

Yes. just paste it into a text file and send it over. It's all the same to me.

[dylanratcliffe]

Here's a video of it happening, I'm pretty convinced that it's just not showing the prompt, since if you hit enter it behaves in the same way as you'd expect if the prompt was there. In this case though I hit Ctrl+C:

Screen.Recording.2024-06-04.at.5.46.02.PM.mp4

stacktrace.txt

[DavidS-ovm]

I think I figured out what this is: interstitialCommand does not correctly implement all required stuff for view freezing. Will have to refactor the surrounding infra a little bit to make this easier on the code, but overall a simple fix.

[DavidS-ovm]

Even after a bunch of debugging, Dylan's VSCode shell is the only instance where we could observe this. Gonna punt on this for now.