Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

investigate flaky test-vm-timeout-escape-promise-module-2 on freebsd #36627

Closed
Trott opened this issue Dec 25, 2020 · 4 comments
Closed

investigate flaky test-vm-timeout-escape-promise-module-2 on freebsd #36627

Trott opened this issue Dec 25, 2020 · 4 comments
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. freebsd Issues and PRs related to the FreeBSD platform.

Comments

@Trott
Copy link
Member

Trott commented Dec 25, 2020

  • Test: parallel/test-vm-timeout-escape-promise-module-2
  • Platform: freebsd
  • Console Output:
20:55:36 not ok 2591 parallel/test-vm-timeout-escape-promise-module-2
20:55:36   ---
20:55:36   duration_ms: 0.397
20:55:36   severity: fail
20:55:36   exitcode: 1
20:55:36   stack: |-
20:55:36     (node:79375) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
20:55:36     (Use `node --trace-warnings ...` to show where the warning was created)
20:55:36     /usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:22
20:55:36           throw new Error(
20:55:36                 ^
20:55:36     
20:55:36     Error: escaped timeout at 173 milliseconds!
20:55:36         at loop (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:22:13)
20:55:36         at vm:module(0):1:30
20:55:36         at SourceTextModule.evaluate (node:internal/vm/module:229:23)
20:55:36         at assert.rejects.code (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:38:16)
20:55:36         at processTicksAndRejections (node:internal/process/task_queues:93:5)
20:55:36   ...
@Trott Trott added freebsd Issues and PRs related to the FreeBSD platform. flaky-test Issues and PRs related to the tests with unstable failures on the CI. labels Dec 25, 2020
@Trott
Copy link
Member Author

Trott commented Mar 20, 2021

https://ci.nodejs.org/job/node-test-commit-freebsd/38718/nodes=freebsd11-x64/console

00:20:50 not ok 2672 parallel/test-vm-timeout-escape-promise-module-2
00:20:51   ---
00:20:51   duration_ms: 0.440
00:20:51   severity: fail
00:20:51   exitcode: 1
00:20:51   stack: |-
00:20:51     (node:46479) ExperimentalWarning: VM Modules is an experimental feature. This feature could change at any time
00:20:51     (Use `node --trace-warnings ...` to show where the warning was created)
00:20:51     /usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:22
00:20:51           throw new Error(
00:20:51                 ^
00:20:51     
00:20:51     Error: escaped timeout at 179 milliseconds!
00:20:51         at loop (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:22:13)
00:20:51         at vm:module(0):1:30
00:20:51         at SourceTextModule.evaluate (node:internal/vm/module:224:23)
00:20:51         at assert.rejects.code (/usr/home/iojs/build/workspace/node-test-commit-freebsd/nodes/freebsd11-x64/test/parallel/test-vm-timeout-escape-promise-module-2.js:38:16)
00:20:51         at processTicksAndRejections (node:internal/process/task_queues:94:5)
00:20:51   ...

@Trott
Copy link
Member Author

Trott commented Mar 20, 2021

My local results suggest that parallelism may help rather than hurt this test but I want to confirm that on CI, because that's counterintuitive, especially when FreeBSD in CI has a history of being our most parallel-sensitive platform (probably having to do with the way the CI host counts processors).

Stress test with no parallelism: https://ci.nodejs.org/job/node-stress-single-test/231/

Stress test with parallelism: https://ci.nodejs.org/job/node-stress-single-test/232/

@Trott
Copy link
Member Author

Trott commented Mar 20, 2021

Stress test with no parallelism: https://ci.nodejs.org/job/node-stress-single-test/231/

Stress test with parallelism: https://ci.nodejs.org/job/node-stress-single-test/232/

Results sort of show it's flaky with parallelism and reliable without it, but I imagine the p value is small. 0 failures in 1000 runs in the first case, and 2 failures in 1000 runs in the second place.

Let's try them both again with 10000 runs to get a little more confidence:

Not parallel: https://ci.nodejs.org/job/node-stress-single-test/233/
Parallel: https://ci.nodejs.org/job/node-stress-single-test/235/

Trott added a commit to Trott/io.js that referenced this issue Mar 20, 2021
Test is reliable in sequential and not in parallel, so move it there.

Fixes: nodejs#36627
@Trott
Copy link
Member Author

Trott commented Mar 20, 2021

Optimistically opened a PR moving it to sequential. #37842

ruyadorno pushed a commit that referenced this issue Mar 24, 2021
Test is reliable in sequential and not in parallel, so move it there.

Fixes: #36627

PR-URL: #37842
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: James M Snell <[email protected]>
targos pushed a commit that referenced this issue May 1, 2021
Test is reliable in sequential and not in parallel, so move it there.

Fixes: #36627

PR-URL: #37842
Reviewed-By: Benjamin Gruenbaum <[email protected]>
Reviewed-By: Antoine du Hamel <[email protected]>
Reviewed-By: Colin Ihrig <[email protected]>
Reviewed-By: James M Snell <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
flaky-test Issues and PRs related to the tests with unstable failures on the CI. freebsd Issues and PRs related to the FreeBSD platform.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant