Task times out on the terminal smoke tests

[aiday-mar]

Smoke tests regularly fails with the error:

   Terminal
      Terminal Editors
        ✔ should update color of the tab
        ✔ should rename the tab
        ✔ should show the panel when the terminal is moved there and close the editor
        ✔ should open a terminal in a new group for open to the side
        ✔ should open a terminal in a new group when the split button is pressed
        1) should create new terminals in the active editor group via command
        2) should create new terminals in the active editor group via plus button
        3) should create a terminal in the editor area by default
      Terminal Input
        Auto replies
          4) should automatically reply to a custom entry
      Terminal Persistence
        detach/attach
          5) should support basic reconnection
          - should persist buffer content
      Terminal Profiles
        6) should launch the default profile
        7) should set the default profile to a contributed one
        8) should use the default contributed profile on panel open and for splitting
        9) should set the default profile
        10) should use the default profile on panel open and for splitting
        11) createWithProfile command should create a terminal with a profile
        12) createWithProfile command should create a terminal with a contributed profile
##[error]The task has timed out.
Finishing: Run smoke tests (Electron)

Example build: https://dev.azure.com/monacotools/Monaco/_build/results?buildId=284839&view=logs&j=7a91b4be-20b5-51ee-04c3-33f03492145f&t=3255846e-ed0b-5948-a22e-d5f1a1fdd237

[deepak1556]

Crash stack shows the pty module was waiting on the exit status of the process

Thread 15
 0  libc.so.6!__wait4 [wait4.c : 27 + 0x26]
    rax = 0xfffffffffffffe00   rdx = 0x0000000000000000
    rcx = 0x00007f54c0a55c7f   rbx = 0x000035a400d049e0
    rsi = 0x00007f542e5086cc   rdi = 0x0000000000009fe1
    rbp = 0x00007f542e5086f0   rsp = 0x00007f542e508690
     r8 = 0x0000000000000000    r9 = 0x00007f542e509700
    r10 = 0x0000000000000000   r11 = 0x0000000000000293
    r12 = 0x00007ffd55c4c7de   r13 = 0x00007ffd55c4c7df
    r14 = 0x00007f542e5091b8   r15 = 0x00007f542e5086cc
    rip = 0x00007f54c0a55c7f
    Found by: given as instruction pointer in context
 1  pty.node!std::__Cr::__thread_proxy<std::__Cr::tuple<std::__Cr::unique_ptr<std::__Cr::__thread_struct, std::__Cr::default_delete<std::__Cr::__thread_struct> >, SetupExitCallback(Napi::Env, Napi::Function, int)::$_1> >(void*) + 0x54
    rbx = 0x000035a400d049e0   rbp = 0x00007f542e5086f0
    rsp = 0x00007f542e5086c0   r12 = 0x00007ffd55c4c7de
    r13 = 0x00007ffd55c4c7df   r14 = 0x00007f542e5091b8
    r15 = 0x00007f542e5086cc   rip = 0x00007f542e5ba694
    Found by: call frame info
 2  libpthread.so.0!start_thread [pthread_create.c : 477 + 0x12]
    rbx = 0x0000000000000000   rbp = 0x0000000000000000
    rsp = 0x00007f542e508700   r12 = 0x00007ffd55c4c7de
    r13 = 0x00007ffd55c4c7df   r14 = 0x00007ffd55c4c7e0
    r15 = 0x00007f542e5087c0   rip = 0x00007f54c1f8f609
    Found by: call frame info
 3  libc.so.6!__GI___clone + 0x43
    rbx = 0x00007f542e509700   rbp = 0x0000000000000000
    rsp = 0x00007f542e5087c0   r12 = 0x00007ffd55c4c7de
    r13 = 0x00007ffd55c4c7df   r14 = 0x00007ffd55c4c7e0
    r15 = 0x00007f542e5087c0   rip = 0x00007f54c0a92353
    Found by: call frame info

While the main thread aborted

Operating system: Linux
                  5.15.0 -1064-azure #73~20.04.1-Ubuntu SMP Mon May 6 09:43:44 UTC 2024 x86_64
CPU: amd64
     family 6 model 106 stepping 6
     4 CPUs

GPU: UNKNOWN

Crash reason:  SIGABRT
Crash address: 0x0
Process uptime: 1 seconds

Thread 0 (crashed)
 0  libc.so.6!gsignal [raise.c : 51 + 0x0]
    rax = 0x0000000000000000   rdx = 0x0000000000000000
    rcx = 0x00007f54c09b600b   rbx = 0x00007f54bfc52f40
    rsi = 0x00007ffd55c4d820   rdi = 0x0000000000000002
    rbp = 0x00007ffd55c4da90   rsp = 0x00007ffd55c4d820
     r8 = 0x0000000000000000    r9 = 0x00007ffd55c4d820
    r10 = 0x0000000000000008   r11 = 0x0000000000000246
    r12 = 0x00007ffd55c51788   r13 = 0x000035a4002266c8
    r14 = 0x0000000000000000   r15 = 0x0000000000000001
    rip = 0x00007f54c09b600b
    Found by: given as instruction pointer in context
 1  libc.so.6!__GI_abort [abort.c : 79 + 0xa]
    rbx = 0x00007f54bfc52f40   rbp = 0x00007ffd55c4da90
    rsp = 0x00007ffd55c4d940   r12 = 0x00007ffd55c51788
    r13 = 0x000035a4002266c8   r14 = 0x0000000000000000
    r15 = 0x0000000000000001   rip = 0x00007f54c0995859
    Found by: call frame info
 2  code-insiders!uv__epoll_ctl_prep [linux.c : 0 + 0x5]
    rbx = 0x0000000000000016   rbp = 0x00007ffd55c4da90
    rsp = 0x00007ffd55c4da70   r12 = 0x00007ffd55c51788
    r13 = 0x000035a4002266c8   r14 = 0x0000000000000000
    r15 = 0x0000000000000001   rip = 0x000055eb02e6887d
    Found by: call frame info
 3  code-insiders!uv__io_poll [linux.c : 1337 + 0x15]
    rbx = 0x000055eb0ae69378   rbp = 0x00007ffd55c51820
    rsp = 0x00007ffd55c4daa0   r12 = 0x00007ffd55c51788
    r13 = 0x000035a4002266c8   r14 = 0x00007ffd55c50ab0
    r15 = 0x0000000000000016   rip = 0x000055eb02e69985
    Found by: call frame info
 4  code-insiders!uv_run [core.c : 447 + 0x8]
    rbx = 0x000055eb0ae69320   rbp = 0x00007ffd55c51890
    rsp = 0x00007ffd55c51830   r12 = 0x0000000000000000
    r13 = 0x000055eb0ae69368   r14 = 0x00007ffd55c51850
    r15 = 0x0000000000000002   rip = 0x000055eb02e59865
    Found by: call frame info
 5  code-insiders!electron::NodeBindings::UvRunOnce() [node_bindings.cc : 896 + 0xa]
    rbx = 0x000035a40022f400   rbp = 0x00007ffd55c51920
    rsp = 0x00007ffd55c518a0   r12 = 0x0000000000000000
    r13 = 0x0000000000000000   r14 = 0x000035a400680200
    r15 = 0x000035a400292370   rip = 0x000055eb02ff3d6e
    Found by: call frame info
 6  code-insiders!base::TaskAnnotator::RunTaskImpl(base::PendingTask&) [callback.h : 156 + 0x3]
    rbx = 0x000035a400257000   rbp = 0x00007ffd55c519b0
    rsp = 0x00007ffd55c51930   r12 = 0x00007f54bfc52c88
    r13 = 0x0000000000000000   r14 = 0x00007ffd55c51978
    r15 = 0x0000000000000000   rip = 0x000055eb062e0d3f
    Found by: call frame info
 7  code-insiders!base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWorkImpl(base::LazyNow*) [task_annotator.h : 90 + 0x8]
    rbx = 0xaaaaaaaaaaaaaaaa   rbp = 0x00007ffd55c51c40
    rsp = 0x00007ffd55c519c0   r12 = 0x000035a400257000
    r13 = 0x000035a4002f4280   r14 = 0x0000000000000000
    r15 = 0x000055eb0ae0bef0   rip = 0x000055eb062fd58d
    Found by: call frame info
 8  code-insiders!non-virtual thunk to base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork() [thread_controller_with_message_pump_impl.cc : 338 + 0xb]
    rbx = 0x00007ffd55c51cd0   rbp = 0x00007ffd55c51cc0
    rsp = 0x00007ffd55c51c50   r12 = 0x000035a4002f4280
    r13 = 0x00007ffd55c51c50   r14 = 0x000035a4002f4368
    r15 = 0x000035a4002f4448   rip = 0x000055eb062fdb7c
    Found by: call frame info
 9  code-insiders!base::MessagePumpDefault::Run(base::MessagePump::Delegate*) [message_pump_default.cc : 40 + 0xb]
    rbx = 0x000035a4002a0a20   rbp = 0x00007ffd55c51d20
    rsp = 0x00007ffd55c51cd0   r12 = 0x00007ffd55c51cd0
    r13 = 0x7fffffffffffffff   r14 = 0x000035a4002f4368
    r15 = 0x000035a4002a0a30   rip = 0x000055eb0629cee8
    Found by: call frame info
10  code-insiders!base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool, base::TimeDelta) [thread_controller_with_message_pump_impl.cc : 641 + 0x6]
    rbx = 0x000035a4002f4280   rbp = 0x00007ffd55c51d70
    rsp = 0x00007ffd55c51d30   r12 = 0x000035a4002f4298
    r13 = 0x7fffffffffffffff   r14 = 0x0000000000000001
    r15 = 0x000035a4002f44a8   rip = 0x000055eb062fe2cd
    Found by: call frame info
11  code-insiders!base::RunLoop::Run(base::Location const&) [run_loop.cc : 134 + 0x12]
    rbx = 0x00007ffd55c51f20   rbp = 0x00007ffd55c51e50
    rsp = 0x00007ffd55c51d80   r12 = 0x0000000000000000
    r13 = 0x00007ffd55c52320   r14 = 0x00007ffd55c51dd0
    r15 = 0x00007ffd55c51f00   rip = 0x000055eb062c3ced
    Found by: call frame info
12  code-insiders!content::UtilityMain(content::MainFunctionParams) [utility_main.cc : 441 + 0x8]
    rbx = 0x00007ffd55c51f00   rbp = 0x00007ffd55c522d0
    rsp = 0x00007ffd55c51e60   r12 = 0xaaaaaaaaaaaaaaaa
    r13 = 0x00007ffd55c52320   r14 = 0x000035a400225c00
    r15 = 0x000035a400330000   rip = 0x000055eb05fc0cd7
    Found by: call frame info
13  code-insiders!content::RunOtherNamedProcessTypeMain(std::__Cr::basic_string<char, std::__Cr::char_traits<char>, std::__Cr::allocator<char>> const&, content::MainFunctionParams, content::ContentMainDelegate*) [content_main_runner_impl.cc : 779 + 0x8]
    rbx = 0x00007ffd55c52390   rbp = 0x00007ffd55c52430
    rsp = 0x00007ffd55c522e0   r12 = 0x000055eb0ab8f360
    r13 = 0x0000000000000000   r14 = 0x00007ffd55c52320
    r15 = 0x00007ffd55c52390   rip = 0x000055eb03183ff0
    Found by: call frame info
14  code-insiders!content::ContentMainRunnerImpl::Run() [content_main_runner_impl.cc : 1158 + 0x8]
    rbx = 0x000035a400290380   rbp = 0x00007ffd55c52550
    rsp = 0x00007ffd55c52440   r12 = 0x000055eb0255e1a0
    r13 = 0xaaaaaaaaaaaaaaaa   r14 = 0x00007ffd55c52478
    r15 = 0x00007ffd55c52400   rip = 0x000055eb03185207
    Found by: call frame info
15  code-insiders!content::RunContentProcess(content::ContentMainParams, content::ContentMainRunner*) [content_main.cc : 331 + 0x8]
    rbx = 0x000035a400290380   rbp = 0x00007ffd55c527b0
    rsp = 0x00007ffd55c52560   r12 = 0x000000000000000e
    r13 = 0x00007ffd55c52750   r14 = 0x0000000000000007
    r15 = 0x00007ffd55c525e1   rip = 0x000055eb03182c2b
    Found by: call frame info
16  code-insiders!content::ContentMain(content::ContentMainParams) [content_main.cc : 344 + 0x8]
    rbx = 0x00007ffd55c52830   rbp = 0x00007ffd55c52820
    rsp = 0x00007ffd55c527c0   r12 = 0x00007ffd55c527e8
    r13 = 0x00007ffd55c529f0   r14 = 0x00007ffd55c527e0
    r15 = 0x00007ffd55c527c8   rip = 0x000055eb03182d30
    Found by: call frame info
17  code-insiders!main [electron_main_linux.cc : 45 + 0x8]
    rbx = 0x00007ffd55c529f8   rbp = 0x00007ffd55c52900
    rsp = 0x00007ffd55c52830   r12 = 0x00007ffd55c528b0
    r13 = 0x00007ffd55c529f0   r14 = 0x00007ffd55c528b0
    r15 = 0x00007ffd55c52830   rip = 0x000055eb02e6ca68
    Found by: call frame info
18  libc.so.6!__libc_start_main [libc-start.c : 308 + 0x1a]
    rbx = 0x000055eb02e05150   rbp = 0x0000000000000000
    rsp = 0x00007ffd55c52910   r12 = 0x000055eb02a63000
    r13 = 0x00007ffd55c529f0   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007f54c0997083
    Found by: call frame info
19  code-insiders!_start + 0x2a
    rbx = 0x0000000000000000   rbp = 0x0000000000000000
    rsp = 0x00007ffd55c529e0   r12 = 0x000055eb02a63000
    r13 = 0x00007ffd55c529f0   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x000055eb02a6302a
    Found by: call frame info
20  0x7ffd55c529e8
    rbx = 0x0000000000000000   rbp = 0x0000000000000000
    rsp = 0x00007ffd55c529e8   r12 = 0x000055eb02a63000
    r13 = 0x00007ffd55c529f0   r14 = 0x0000000000000000
    r15 = 0x0000000000000000   rip = 0x00007ffd55c529e8
    Found by: call frame info

[deepak1556]

I am not sure what could have caused this state, but the only change that might be relevant was electron/electron@f3cc8cb which came in with the Electron update 2 days ago #222756. I will revert the update for now since I will be out next week and revisit this in August.

[deepak1556]

Change has been reverted, adjusting labels accordingly. I will keep this issue open to track the root cause and bring back the update in August.

[Tyriar]

Still failing with the revert

[deepak1556]

CI runs have been stable after triggering an insiders with the reverted change, so maybe it was a false alarm. Will watch for the next failure.

PR pipeline failed on the reverted commit https://monacotools.visualstudio.com/Monaco/_build/results?buildId=284901&view=results
While insiders pipeline passed on the same commit which ran after https://monacotools.visualstudio.com/Monaco/_build/results?buildId=284903&view=results

Both use the same linux build agents and both of them did recompile the native modules, there should be no difference in module headers since they use the same electron version. Given the failure was linux specific, I was wondering if io_uring got reenabled since we know pty module does not work well in this mode. But I don't see any changes in the runtime related, also given only smoke tests are affected points that might not be the case.

@Tyriar if you have the chance please try the smoke tests on the commit before the revert locally and let me know if the failures are consistent.

[deepak1556]

Merging to #216564