Out of memory in integration tests

[mikemurray]

Out of memory in jest integration tests. This the error I get bellow after reaction ~2gb of memory used. Each test increases the heap size which suggests a memory leak.

(node:96982) DeprecationWarning: collection.find option [fields] is deprecated and will be removed in a later version.
(node:96982) DeprecationWarning: collection.find option [fields] is deprecated and will be removed in a later version.
 PASS  tests/integration/api/queries/catalogItems/catalogItemsByTag.test.js (13.152s, 2018 MB heap size)

 RUNS  tests/integration/api/mutations/addAccountEmailRecord/addAccountEmailRecord.test.js

<--- Last few GCs --->

[96982:0x104059000]   489172 ms: Scavenge 2014.2 (2053.0) -> 2013.3 (2053.0) MB, 12.2 / 0.1 ms  (average mu = 0.162, current mu = 0.163) allocation failure
[96982:0x104059000]   489203 ms: Scavenge 2014.2 (2053.0) -> 2013.3 (2053.0) MB, 6.1 / 0.1 ms  (average mu = 0.162, current mu = 0.163) allocation failure
[96982:0x104059000]   489238 ms: Scavenge 2014.3 (2053.0) -> 2013.5 (2053.0) MB, 9.5 / 0.1 ms  (average mu = 0.162, current mu = 0.163) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x10093f4f9]
    1: StubFrame [pc: 0x1008ea75f]
Security context: 0x0b715bdc08d1 <JSObject>
    2: resolveStubModuleName [0xb71ef6f41d9] [/Users/mikemurray/Projects/reaction/reaction-platform/reaction/node_modules/jest-resolve/build/index.js:~400] [pc=0x9e57b1825bc](this=0x0b7127861f49 <Resolver map = 0xb71d38763f9>,0x0b71e77f9059 <String[#109]: /Users/mikemurray/Projects/reaction/reaction-platform/reaction/src/plugi...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Writing Node.js report to file: report.20191211.095820.96982.0.001.json
Node.js report completed
 1: 0x10007f3b9 node::Abort() [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
 2: 0x10007f53d node::OnFatalError(char const*, char const*) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
 3: 0x100173617 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
 4: 0x1001735b3 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
 5: 0x1002f9815 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
 6: 0x1002faee4 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
 7: 0x1002f7d57 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
 8: 0x1002f5d26 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
 9: 0x10030153a v8::internal::Heap::AllocateRawWithLightRetry(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
10: 0x1003015c1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
11: 0x1002ce73b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
12: 0x1006076f8 v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
13: 0x10093f4f9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
14: 0x1008ea75f Builtins_CloneFastJSArray [/Users/mikemurray/.nvm/versions/node/v13.0.1/bin/node]
[1]    96981 abort      npm run test:integration:heap

Possible Solution

Perhaps the a new in memory mongo instance is being carried over from each test?

[trojanh]

@mikemurray after using --clearCache I could see the integration tests start with an initial heap size of 169 MB instead of 411 MB so this gives room for some more tests to run but doesn’t solve the problem of increasing heap size.

I tried the following things:

• clearing jest cache
• used await for testApp.stop() where it was not being used, so each test file waits for the connection to close before moving to the next tests file.
• tested if mongoClient.close(), gets invoked after every test run.
• verified if we force close mongodb connection before starting a new one here

These solutions don't seem to work. Then I found that similar issue was already created => jestjs/jest#9081

You have anything else in mind that I could debug or I could help you solve this?

[trojanh]

Just noticed this issue occured on circleci for this PR of mine.

[mikemurray]

@trojanh, For now, I've split the query & mutation integration tests to two jobs on CI each with a 4gb memory limit. This will buy us some time to figure out a proper solution.