JavaScript heap out of memory with an unreleased npm package

[mansona]

Description

I just tried to use np to release a package that hadn't been released yet but I got a JavaScript heap out of memory error with a very fun looking stack trace 😂

➜  ember-deprecation-error git:(master) ✗ np

<--- Last few GCs --->

[55609:0x104907000]    11975 ms: Mark-sweep (reduce) 4093.4 (4112.0) -> 4093.4 (4113.0) MB, 13.1 / 0.0 ms  (average mu = 0.201, current mu = 0.001) last resort GC in old space requested
[55609:0x104907000]    11988 ms: Mark-sweep (reduce) 4093.4 (4112.0) -> 4093.4 (4113.0) MB, 13.0 / 0.0 ms  (average mu = 0.112, current mu = 0.001) last resort GC in old space requested


<--- JS stacktrace --->

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0x101312195 node::Abort() (.cold.1) [/usr/local/bin/node]
 2: 0x1000ade29 node::Abort() [/usr/local/bin/node]
 3: 0x1000adf8f node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 4: 0x1001f2407 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x1001f23a3 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 6: 0x1003a1fb5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 7: 0x1003ab3ec v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 8: 0x100373a9d v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [/usr/local/bin/node]
 9: 0x10060a0c1 v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::AllocationType) [/usr/local/bin/node]
10: 0x100720850 v8::internal::Runtime_StringSplit(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
11: 0x100a85499 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
[1]    55609 abort      np

Steps to reproduce

I'm pretty sure the only special thing that I was doing was trying to release a package that hadn't been released yet 🤔 I released using npm publish and then ran np again and it gave me the No commits found since previous release, continue? warning so I figured it was back working

Environment

np - latest (I re-installed after i saw the error first and it still happened)
Node.js - I started on node 12 but then I installed a fresh Node 14 system-wide v14.17.6
npm - 7.22.0
Git - git version 2.30.1 (Apple Git-130)
OS - macOs 11.5.2 (20G95)

[dopecodez]

That is a very weird error, I have not seen this anywhere I've used np on my Mac.

I tried multiple times releasing new packages with np because of #613 but this seems to be something else completely. The V8 engine running out of memory for a package might be because of package size or some other weird behaviour with np.

We'll keep this ticket open to see if the community runs into this anywhere.

[mansona]

Do it wasn't a large package, unless it was somehow including node modules which is very strange behaviour.

I noticed something after I posted this bug: it wasn't just unreleased, I had also forgotten to create a remote repo

[dopecodez]

I had also forgotten to create a remote repo

This would definitely cause np to error, but still, your error sounds weird. Remote does not exist is handled within np, your case seems to be something else entirely.

[mansona]

So I just came across this again with a repo that I literally just released earlier today, so the root cause isn't the missing remote 🤔

New stack trace:

➜  ember-cli-showdown git:(master) np

<--- Last few GCs --->

[78017:0x104907000]    13476 ms: Mark-sweep (reduce) 4093.7 (4110.9) -> 4093.7 (4111.9) MB, 18.4 / 0.0 ms  (average mu = 0.190, current mu = 0.001) last resort GC in old space requested
[78017:0x104907000]    13495 ms: Mark-sweep (reduce) 4093.7 (4110.9) -> 4093.7 (4111.9) MB, 18.9 / 0.0 ms  (average mu = 0.104, current mu = 0.001) last resort GC in old space requested


<--- JS stacktrace --->

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: 0x101312195 node::Abort() (.cold.1) [/usr/local/bin/node]
 2: 0x1000ade29 node::Abort() [/usr/local/bin/node]
 3: 0x1000adf8f node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 4: 0x1001f2407 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x1001f23a3 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 6: 0x1003a1fb5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 7: 0x1003ab3ec v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
 8: 0x100373a9d v8::internal::FactoryBase<v8::internal::Factory>::NewRawOneByteString(int, v8::internal::AllocationType) [/usr/local/bin/node]
 9: 0x10060a0c1 v8::internal::String::SlowFlatten(v8::internal::Isolate*, v8::internal::Handle<v8::internal::ConsString>, v8::internal::AllocationType) [/usr/local/bin/node]
10: 0x100720850 v8::internal::Runtime_StringSplit(int, unsigned long*, v8::internal::Isolate*) [/usr/local/bin/node]
11: 0x100a85499 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/bin/node]
[1]    78017 abort      np

[mansona]

I just found the issue!

I debugged using node --inspect-brk /usr/local/bin/np and apparently the debugger pauses "before potential out-of-memory crash" which is very handy 🎉

The issue is that I have a folder locally that is a github cache for auto-changelog. This file is ignored with .gitignore and I just tested it with .npmignore too and it is dying because of that folder.

The first thing in the stacktrace that is your code is this:

async function getFilesIgnoredByDotnpmignore(pkg, fileList) {
	const allowList = await ignoreWalker({
		path: pkgDir.sync(),
		ignoreFiles: ['.npmignore']
	});
	return fileList.filter(minimatch.filter(getIgnoredFilesGlob(allowList, pkg.directories), {matchBase: true, dot: true}));
}

and allowList has a length of 41147 😱

[mansona]

In fact I needed to delete node_modules as well before this would work 🙈

[mansona]

Just a bump on this issue 👋

Does anyone have any pointers on how I should proceed with a fix for this? I don't know too much about the internals of this so I got a bit stuck after investigating the "cause". Any pointers or suggestions would be very much appreciated as I'm still hitting this nearly every time I'm running np

[dopecodez]

sorry for the delayed response @mansona but I still am unable to reproduce this error on my local, it will be very helpful if you can provide a reproduction repo for the same.

[mansona]

I have a feeling that #633 might solve this particular issue 🤔 not exactly sure but it seems like it might be related

I'll keep y'all posted to see if I come across it again since that has been released 👍

[fregante]

Closing as #633 likely fixed it