apex get test --codecoverage runs 30x times slower or crashes with JavaScript heap out of memory

[maksym-petrov-ct]

Summary

Compared to sfdx 7.x versions, sf versions (tested on 2.52 and 2.68.6) can't effectively run sf apex get test --codecoverage on org with hundreds of class files. It either crashes after ~30 min, or if successful, takes ~30 min. sfdx 7.x with force:apex:test:report --codecoverage takes 1-2 min and never crashes here.

Expected result

1. generate test results with code coverage (works as expected):

sf apex run test --wait 1 --result-format json --code-coverage --json -o TargetOrg --test-level RunSpecifiedTests --tests "Test_1,Test_2..."

2. after tests have fully finished, I'm getting test-run-id and running below command:

sf apex get test -o TargetOrg --code-coverage --test-run-id 70700000000000000 --json  --outputdir ./test_results > stdout.txt

in result I expect in 1-2 min to get test_results directory containing files:

test-result.txt
test-result-70700000000000000.json
test-result-70700000000000000-codecoverage.json
test-result-70700000000000000-junit.xml
test-result-codecoverage.json
test-run-id.txt

note that I compared behavior with sfdx v7.180 and sfdx gets job done with it's force:apex:test:report in 1-2 min.

Actual result

after having code coverage stats in org I can get it's test-run-id, something like 7070000000000000X. Some tests have failed in this test-run and overall coverage is below 75%, if it matters (but I think it doesn't).

Note that sf command is passed through 'time' and below is stats for system resources usage during execution (29 min of real time while only 1m23s of active time).

time sf apex get test -o TargetOrg --code-coverage --test-run-id 70700000000000000 --json  --outputdir ./test_results > stdout.txt

<--- Last few GCs --->

[508145:0x35ea3800]  1734278 ms: Scavenge 2033.6 (2071.1) -> 2030.4 (2072.1) MB, 3.74 / 0.00 ms  (average mu = 0.192, current mu = 0.131) allocation failure; 
[508145:0x35ea3800]  1734286 ms: Scavenge 2034.6 (2072.1) -> 2031.5 (2073.4) MB, 4.07 / 0.00 ms  (average mu = 0.192, current mu = 0.131) allocation failure; 
[508145:0x35ea3800]  1734292 ms: Scavenge 2036.1 (2073.6) -> 2033.0 (2074.9) MB, 3.61 / 0.00 ms  (average mu = 0.192, current mu = 0.131) allocation failure; 


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0xb82c28 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [node]
 2: 0xeed540 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 3: 0xeed827 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [node]
 4: 0x10ff3c5  [node]
 5: 0x10ff954 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 6: 0x1116844 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const*) [node]
 7: 0x111705c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 8: 0x11191ba v8::internal::Heap::HandleGCRequest() [node]
 9: 0x1084827 v8::internal::StackGuard::HandleInterrupts() [node]
10: 0x1526f7a v8::internal::Runtime_StackGuard(int, unsigned long*, v8::internal::Isolate*) [node]
11: 0x7fca2f699ef6 
Aborted (core dumped)

real    29m2.633s
user    1m23.516s
sys     0m9.384s

Additional information

after ~30 min sometimes test_results directory contains expected files, but sometimes not.
Reproduced on MacOS, Ubuntu. Tested on version @salesforce/cli/2.52 - issue reproduces.

System Information

{
  "architecture": "wsl-x64",
  "cliVersion": "@salesforce/cli/2.68.6",
  "nodeVersion": "node-v20.13.1",
  "osVersion": "Linux 5.15.167.4-microsoft-standard-WSL2",
  "rootPath": "/home/ubuntu/.nvm/versions/node/v20.13.1/lib/node_modules/@salesforce/cli",
  "shell": "bash",
  "pluginVersions": [
    "@oclif/plugin-autocomplete 3.2.10 (core)",
    "@oclif/plugin-commands 4.1.10 (core)",
    "@oclif/plugin-help 6.2.18 (core)",
    "@oclif/plugin-not-found 3.2.28 (core)",
    "@oclif/plugin-plugins 5.4.17 (core)",
    "@oclif/plugin-search 1.2.16 (core)",
    "@oclif/plugin-update 4.6.13 (core)",
    "@oclif/plugin-version 2.2.16 (core)",
    "@oclif/plugin-warn-if-update-available 3.1.23 (core)",
    "@oclif/plugin-which 3.2.19 (core)",
    "@salesforce/cli 2.68.6 (core)",
    "@salesforce/commerce 252.0.0 (user) published 181 days ago (Thu Jun 13 2024)",
    "apex 3.6.3 (core)",
    "api 1.3.2 (core)",
    "auth 3.6.75 (core)",
    "community 3.2.32 (user) published 88 days ago (Sat Sep 14 2024) (latest is 3.3.6)",
    "data 3.11.4 (core)",
    "deploy-retrieve 3.15.13 (core)",
    "info 3.4.21 (core)",
    "limits 3.3.40 (core)",
    "marketplace 1.3.6 (core)",
    "org 5.2.4 (core)",
    "packaging 2.9.3 (core)",
    "schema 3.3.42 (core)",
    "settings 2.4.6 (core)",
    "sobject 1.4.46 (core)",
    "telemetry 3.6.23 (core)",
    "templates 56.3.30 (core)",
    "trust 3.7.43 (core)",
    "user 3.6.3 (core)",
    "@salesforce/sfdx-scanner 4.2.0 (user) published 197 days ago (Tue May 28 2024) (latest is 4.7.0)",
    "sfdx-git-delta 5.49.3 (user) published 29 days ago (Tue Nov 12 2024) (latest is 5.50.0)"
  ]
}

[github-actions]

Thank you for filing this issue. We appreciate your feedback and will review the issue as soon as possible. Remember, however, that GitHub isn't a mechanism for receiving support under any agreement or SLA. If you require immediate assistance, contact Salesforce Customer Support.

[iowillhoit]

Hey @maksym-petrov-ct, thank you for providing all of these details. I have tried to reproduce your findings using my https://github.com/iowillhoit/dreamhouse-lots-o-tests repo. I created 3800 "fast" tests (which is 97.89% of the Apex character limit in my scratch org) and executed them using your commands.

The tests themselves took a long time to execute, but the get command only took 27 seconds:
sf apex get test --code-coverage --test-run-id 7078N0000JPeNmlQQF --json --outputdir ./test_results > stdout.txt

I tried this again using sfdx version 7.209.6 and it finished in 21 seconds:
sfdx force:apex:test:report --code-coverage --test-run-id 7078N0000JPeNmlQQF --json --outputdir ./test_results > stdout.txt

A few things:

• Are you able to get us access to an org where we could try to reproduce this? (we could email)
• The team that owns the apex library has re-worked the formatters to help with some of out-of-memory issues. The CLI team has not yet implemented them. For internal tracking, this is the work item: W-16959423
• In the meantime and as a workaround, we generally recommend running subsets of tests using Test Suites.

[maksym-petrov-ct]

Hi @iowillhoit
thanks for investigating this.
Yes, I believe I can provide access to one of sandboxes.
would the route through Salesforce support case, issued from production work here ?

[iowillhoit]

Cool that might be helpful. For what it is worth, your login might not be 100% necessary since we already have a similar case with some details in it. If you wanted to email the creds, I could give you our team's email address.

Hopefully we will be able to get to this work in January.

[maksym-petrov-ct]

Thanks @iowillhoit. Let's see how it will go in January, please let me know if you still need sandbox to reproduce.