Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_runner: improve enqueue and dequeue #49751

Closed
wants to merge 3 commits into from

Conversation

MoLow
Copy link
Member

@MoLow MoLow commented Sep 21, 2023

extracting parts out of #48932 to land first, will make it easier for me to complete that big PR

@nodejs-github-bot
Copy link
Collaborator

Review requested:

  • @nodejs/test_runner

@nodejs-github-bot nodejs-github-bot added needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem. labels Sep 21, 2023
@MoLow
Copy link
Member Author

MoLow commented Sep 21, 2023

@MoLow MoLow marked this pull request as ready for review September 21, 2023 19:17
@rluvaton
Copy link
Member

Benchmark output:

                                                                                                   confidence improvement accuracy (*)   (**)   (***)
test_runner/global-concurrent-tests.js type='async' n=100                                                  **     -5.62 %       ±3.69% ±4.92%  ±6.40%
test_runner/global-concurrent-tests.js type='async' n=1000                                                        -3.01 %       ±6.33% ±8.43% ±10.97%
test_runner/global-concurrent-tests.js type='async' n=10000                                               ***    -11.68 %       ±1.86% ±2.48%  ±3.24%
test_runner/global-concurrent-tests.js type='sync' n=100                                                   **     -8.29 %       ±5.45% ±7.25%  ±9.46%
test_runner/global-concurrent-tests.js type='sync' n=1000                                                         -3.87 %       ±5.29% ±7.04%  ±9.17%
test_runner/global-concurrent-tests.js type='sync' n=10000                                                ***    -12.91 %       ±1.85% ±2.46%  ±3.21%
test_runner/global-sequential-tests.js type='async' n=100                                                         -0.55 %       ±1.62% ±2.16%  ±2.81%
test_runner/global-sequential-tests.js type='async' n=1000                                                         0.71 %       ±2.49% ±3.32%  ±4.33%
test_runner/global-sequential-tests.js type='async' n=10000                                                        0.68 %       ±1.50% ±2.00%  ±2.62%
test_runner/global-sequential-tests.js type='sync' n=100                                                           0.01 %       ±1.30% ±1.74%  ±2.26%
test_runner/global-sequential-tests.js type='sync' n=1000                                                          1.20 %       ±1.40% ±1.86%  ±2.42%
test_runner/global-sequential-tests.js type='sync' n=10000                                                         1.13 %       ±2.15% ±2.88%  ±3.77%
test_runner/multi-files.js concurrency='no' patterns='test/fixtures/test-runner/**/*.?(c|m)js'                     0.02 %       ±0.64% ±0.85%  ±1.11%
test_runner/multi-files.js concurrency='no' patterns='test/parallel/test-runner-*'                                -0.37 %       ±0.99% ±1.32%  ±1.72%
test_runner/multi-files.js concurrency='yes' patterns='test/fixtures/test-runner/**/*.?(c|m)js'                   -0.06 %       ±0.45% ±0.60%  ±0.78%
test_runner/multi-files.js concurrency='yes' patterns='test/parallel/test-runner-*'                                0.80 %       ±4.69% ±6.29%  ±8.29%
test_runner/suite-tests.js concurrency='no' testType='async' testsPerSuite=10 numberOfSuites=10           ***    -16.52 %       ±7.34% ±9.79% ±12.79%
test_runner/suite-tests.js concurrency='no' testType='async' testsPerSuite=10 numberOfSuites=100                  -5.35 %       ±6.64% ±8.83% ±11.49%
test_runner/suite-tests.js concurrency='no' testType='async' testsPerSuite=100 numberOfSuites=10            *     -7.93 %       ±6.35% ±8.46% ±11.03%
test_runner/suite-tests.js concurrency='no' testType='async' testsPerSuite=100 numberOfSuites=100         ***    -15.74 %       ±3.83% ±5.12%  ±6.69%
test_runner/suite-tests.js concurrency='no' testType='sync' testsPerSuite=10 numberOfSuites=10             **     -9.80 %       ±5.75% ±7.66%  ±9.97%
test_runner/suite-tests.js concurrency='no' testType='sync' testsPerSuite=10 numberOfSuites=100                   -6.93 %       ±7.02% ±9.34% ±12.16%
test_runner/suite-tests.js concurrency='no' testType='sync' testsPerSuite=100 numberOfSuites=10            **    -10.70 %       ±6.46% ±8.62% ±11.27%
test_runner/suite-tests.js concurrency='no' testType='sync' testsPerSuite=100 numberOfSuites=100          ***    -15.67 %       ±4.25% ±5.67%  ±7.40%
test_runner/suite-tests.js concurrency='yes' testType='async' testsPerSuite=10 numberOfSuites=10           **     -9.35 %       ±5.73% ±7.62%  ±9.92%
test_runner/suite-tests.js concurrency='yes' testType='async' testsPerSuite=10 numberOfSuites=100                 -3.61 %       ±7.13% ±9.49% ±12.35%
test_runner/suite-tests.js concurrency='yes' testType='async' testsPerSuite=100 numberOfSuites=10                 -5.42 %       ±6.65% ±8.86% ±11.53%
test_runner/suite-tests.js concurrency='yes' testType='async' testsPerSuite=100 numberOfSuites=100        ***    -15.05 %       ±4.11% ±5.48%  ±7.14%
test_runner/suite-tests.js concurrency='yes' testType='sync' testsPerSuite=10 numberOfSuites=10           ***    -15.59 %       ±6.39% ±8.52% ±11.13%
test_runner/suite-tests.js concurrency='yes' testType='sync' testsPerSuite=10 numberOfSuites=100                  -3.53 %       ±6.85% ±9.12% ±11.87%
test_runner/suite-tests.js concurrency='yes' testType='sync' testsPerSuite=100 numberOfSuites=10                  -2.89 %       ±3.45% ±4.63%  ±6.11%
test_runner/suite-tests.js concurrency='yes' testType='sync' testsPerSuite=100 numberOfSuites=100         ***    -14.85 %       ±4.63% ±6.17%  ±8.04%

Be aware that when doing many comparisons the risk of a false-positive
result increases. In this case, there are 32 comparisons, you can thus
expect the following amount of false-positive results:
  1.60 false positives, when considering a   5% risk acceptance (*, **, ***),
  0.32 false positives, when considering a   1% risk acceptance (**, ***),
  0.03 false positives, when considering a 0.1% risk acceptance (***)

@RelxOff
Copy link

RelxOff commented Sep 22, 2023

💯

@MoLow MoLow force-pushed the test-runner-enqueu-dequeue branch from 6626cd4 to 37e1373 Compare September 23, 2023 20:16
@MoLow MoLow closed this Mar 17, 2024
@MoLow MoLow deleted the test-runner-enqueu-dequeue branch May 24, 2024 09:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-ci PRs that need a full CI run. test_runner Issues and PRs related to the test runner subsystem.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants