sequential.test-watch-mode-inspect seems flaky #49933

mhdawson · 2023-09-28T15:59:36Z

Test

sequential.test-watch-mode-inspect

Platform

Linux ARMv7

Console output

Error Message
fail (-15)
Stacktrace
---
duration_ms: 360029.938
exitcode: -15
severity: fail
...

Build links

Additional information

No response

mhdawson · 2023-09-28T16:01:52Z

Failed on same machine, two builds in a row
test-equinix-ubuntu2004_container-armv7l-1

mhdawson · 2023-09-28T16:54:45Z

Passed when it ran on test-equinix-ubuntu2204-x64-2 so could be specific to that machine.

Need to track to see if it fails across the machines or not.

mhdawson · 2023-09-29T14:40:24Z

and again - https://ci.nodejs.org/job/node-test-binary-armv7l/7720/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/console

mhdawson · 2023-09-29T14:41:00Z

Interestingly I see this in the output

not ok 3844 sequential/test-watch-mode-inspect # TODO : Fix flaky test

which seems to say it's a known issue even though I could not find a tracking issue.

richardlau · 2023-09-29T14:53:58Z

FWIW the test is already marked flaky

node/test/sequential/sequential.status

Lines 10 to 12 in 1b96975

    
           # https://github.com/nodejs/node/issues/44898 
        
           test-watch-mode: PASS, FLAKY 
        
           test-watch-mode-inspect: PASS, FLAKY

https://ci.nodejs.org/job/node-test-binary-armv7l/7720/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/console has failed because of leftover Node.js processes (which are probably related to the test being timed out):

17:02:09 All tests passed.
17:02:09 ps awwx | grep Release/node | grep -v grep | cat
17:02:09 2212632 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 --watch /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
17:02:09 2212638 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
17:02:09 make: *** [Makefile:538: test-ci-js] Error 1

anonrig · 2023-10-17T22:08:29Z

It failed on again on ubuntu 22.04 https://ci.nodejs.org/job/node-test-commit-osx-arm/13900/

anonrig · 2023-10-18T21:27:53Z

Seen on node-test-commit-arm-fanned https://ci.nodejs.org/job/node-test-binary-armv7l/8168/

mhdawson · 2023-10-20T16:05:16Z

I don't see if failing in https://ci.nodejs.org/job/node-test-commit-osx-arm/13900/, that seemed to be a different test test-watch-mode without the inspect part.

PR-URL: #50259 Refs: #49933 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>

PR-URL: nodejs#50259 Refs: nodejs#49933 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>

PR-URL: #50259 Refs: #49933 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>

mhdawson · 2023-11-23T15:03:06Z

again - https://ci.nodejs.org/job/node-test-binary-armv7l/8829/

PR-URL: #50259 Refs: #49933 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>

mhdawson · 2023-12-04T20:38:02Z

again - https://ci.nodejs.org/job/node-test-binary-armv7l/9064/

mhdawson · 2023-12-04T20:52:07Z

Not sure why I keep seeing these even though the test is marked as flaky. Looked at the CI jobs and I don't see anything obviously wrong.

@richardlau do you know of any reason off the top of your head a job might not be treating flaky tests as ok?

richardlau · 2023-12-06T13:30:30Z

@mhdawson https://ci.nodejs.org/job/node-test-binary-armv7l/9064/RUN_SUBSET=js,nodes=ubuntu2004-armv7l/console failed because:

18:27:41 All tests passed.
18:27:41 ps awwx | grep Release/node | grep -v grep | cat
18:27:41  601756 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 --watch /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
18:27:41  601762 ?        Sl     0:00 /home/iojs/build/workspace/node-test-binary-armv7l/out/Release/node --inspect=0 /home/iojs/build/workspace/node-test-binary-armv7l/test/fixtures/watch-mode/inspect.js
18:27:41 make: *** [Makefile:538: test-ci-js] Error 1

"All tests passed" means that the failure was treated as flaky. The problem is that it left behind child processes and the build will fail if any node process are still running after the test runner exits:

node/Makefile

Lines 554 to 559 in 342ddb0

    
           $(info Clean up any leftover processes, error if found.) 
        
           ps awwx | grep Release/node | grep -v grep | cat 
        
           @PS_OUT=`ps awwx | grep Release/node | grep -v grep | awk '{print $$1}'`; \ 
        
           if [ "$${PS_OUT}" ]; then \ 
        
           	echo $${PS_OUT} | xargs kill -9; exit 1; \ 
        
           fi

mhdawson · 2023-12-07T15:25:59Z

@richardlau, if I get what you are saying is that even though the job out reports this test, that was not the problem, instead it was because child processes were left behind that may/likely have nothing to do with that flaky test right?

richardlau · 2023-12-07T15:37:08Z

@mhdawson Yes, although I suspect the leftover processes are from the flaky test which I believe has been timed out.

mhdawson · 2023-12-07T15:51:16Z

@richardlau ok, if we believe they are related to the test then SKIP might be more appropriate right? If so I'll submit a PR to do that.

mhdawson · 2023-12-18T18:53:53Z

Failed again - https://ci.nodejs.org/job/node-test-binary-armv7l/9322/

Refs: nodejs#49933 - SKIP instead of mark FLAKY as tests seems to fail builds through left over processe Signed-off-by: Michael Dawson <[email protected]>

Refs: #49933 - SKIP instead of mark FLAKY as tests seems to fail builds through left over processe Signed-off-by: Michael Dawson <[email protected]> PR-URL: #51210 Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: James M Snell <[email protected]>

mhdawson · 2024-01-02T22:29:53Z

Faillures on ubuntu shared libs - https://ci.nodejs.org/job/node-test-commit-linux-containered/40987/nodes=ubuntu2204_sharedlibs_shared_x64/testReport/junit/(root)/sequential/test_watch_mode_inspect_/

mhdawson · 2024-02-02T15:21:37Z

Seems to fail on ubuntu as well - https://ci.nodejs.org/job/node-test-commit-linux-containered/nodes=ubuntu2204_sharedlibs_shared_x64/41479/

Refs: #49933 - SKIP instead of mark FLAKY as tests seems to fail builds through left over processe Signed-off-by: Michael Dawson <[email protected]> PR-URL: #51210 Reviewed-By: Luigi Pinca <[email protected]> Reviewed-By: James M Snell <[email protected]>

PR-URL: nodejs/node#50259 Refs: nodejs/node#49933 Reviewed-By: Moshe Atlow <[email protected]> Reviewed-By: Geoffrey Booth <[email protected]> Reviewed-By: Michael Dawson <[email protected]> Reviewed-By: Vinícius Lourenço Claro Cardoso <[email protected]>

jakecastelli · 2024-07-30T04:08:25Z

Again - https://ci.nodejs.org/job/node-test-commit-smartos/nodes=smartos20-64/55713/

logs

22:09:53 not ok 4228 sequential/test-watch-mode-inspect # TODO : Fix flaky test
22:09:53   ---
22:09:53   duration_ms: 300129.29800
22:09:53   severity: fail
22:09:53   exitcode: -15
22:09:53   stack: |-
22:09:53     timeout
22:09:53     [test] Connecting to a child Node process
22:09:53     [test] Testing /json/list
22:09:53     TAP version 13
22:09:53     [err] Debugger listening on ws://127.0.0.1:61622/bec85c05-bcf7-4407-86de-720d30c10727
22:09:53     [err] For help, see: https://nodejs.org/en/docs/inspector
22:09:53     [err] 
22:09:53     [err] Debugger attached.
22:09:53     [err] 
22:09:53     [test] Connecting to a child Node process
22:09:53     [test] Testing /json/list
22:09:53     [err] Debugger ending on ws://127.0.0.1:61622/bec85c05-bcf7-4407-86de-720d30c10727
22:09:53     [err] For help, see: https://nodejs.org/en/docs/inspector
22:09:53     [err] 
22:09:53     [out] safe to debug now
22:09:53     [out] 
22:09:53   ...

This test seems to fail quite often lately

targos · 2024-08-08T13:26:53Z

https://ci.nodejs.org/job/node-test-commit-smartos/55913/nodes=smartos20-64/consoleFull

aduh95 · 2024-09-10T09:14:58Z

Again https://ci.nodejs.org/job/node-test-commit-linux-containered/46199/

mhdawson added the flaky-test Issues and PRs related to the tests with unstable failures on the CI. label Sep 28, 2023

github-actions bot added the arm Issues and PRs related to the ARM platform. label Sep 28, 2023

anonrig mentioned this issue Oct 18, 2023

test: set test-watch-mode-inspect as flaky #50259

Merged

mhdawson added a commit to mhdawson/io.js that referenced this issue Dec 18, 2023

test: skip test-watch-mode-inspect on arm

64f8f28

Refs: nodejs#49933 - SKIP instead of mark FLAKY as tests seems to fail builds through left over processe Signed-off-by: Michael Dawson <[email protected]>

mhdawson mentioned this issue Dec 18, 2023

test: skip test-watch-mode-inspect on arm #51210

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

sequential.test-watch-mode-inspect seems flaky #49933

sequential.test-watch-mode-inspect seems flaky #49933

mhdawson commented Sep 28, 2023 •

edited

Loading

mhdawson commented Sep 28, 2023

mhdawson commented Sep 28, 2023

mhdawson commented Sep 29, 2023

mhdawson commented Sep 29, 2023

richardlau commented Sep 29, 2023

anonrig commented Oct 17, 2023

anonrig commented Oct 18, 2023

mhdawson commented Oct 20, 2023

mhdawson commented Nov 23, 2023

mhdawson commented Dec 4, 2023 •

edited

Loading

mhdawson commented Dec 4, 2023

richardlau commented Dec 6, 2023

mhdawson commented Dec 7, 2023

richardlau commented Dec 7, 2023

mhdawson commented Dec 7, 2023

mhdawson commented Dec 18, 2023

mhdawson commented Jan 2, 2024

mhdawson commented Feb 2, 2024

jakecastelli commented Jul 30, 2024

targos commented Aug 8, 2024

aduh95 commented Sep 10, 2024

sequential.test-watch-mode-inspect seems flaky #49933

sequential.test-watch-mode-inspect seems flaky #49933

Comments

mhdawson commented Sep 28, 2023 • edited Loading

Test

Platform

Console output

Build links

Additional information

mhdawson commented Sep 28, 2023

mhdawson commented Sep 28, 2023

mhdawson commented Sep 29, 2023

mhdawson commented Sep 29, 2023

richardlau commented Sep 29, 2023

anonrig commented Oct 17, 2023

anonrig commented Oct 18, 2023

mhdawson commented Oct 20, 2023

mhdawson commented Nov 23, 2023

mhdawson commented Dec 4, 2023 • edited Loading

mhdawson commented Dec 4, 2023

richardlau commented Dec 6, 2023

mhdawson commented Dec 7, 2023

richardlau commented Dec 7, 2023

mhdawson commented Dec 7, 2023

mhdawson commented Dec 18, 2023

mhdawson commented Jan 2, 2024

mhdawson commented Feb 2, 2024

jakecastelli commented Jul 30, 2024

targos commented Aug 8, 2024

aduh95 commented Sep 10, 2024

mhdawson commented Sep 28, 2023 •

edited

Loading

mhdawson commented Dec 4, 2023 •

edited

Loading