Segmentation fault in taking a heap snapshot

[lomaster1]

• Version: v10.16.3 (we're using nave)
• Platform: Linux zagent 4.4.0-159-generic configure: remove workarounds for GCC < 4.8 #187-Ubuntu SMP Thu Aug 1 16:28:06 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
• Subsystem: http2

I have been making a lot of requests to the service (like in #29902) and after that I send SIGUSR2 to the service process to take heap snapshot. And service crashed with a segmentation fault.

Core was generated by `/var/lib/nave/installed/10.16.3/bin/node --expose-internals --expose-gc --max-h'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000001a66b35 in node::MemoryRetainerNode::MemoryRetainerNode (this=0x6e95f10, tracker=0x7ffc9b58b9e0, retainer=0xb3be530) at ../src/memory_tracker-inl.h:29
29	    v8::Local<v8::Object> obj = retainer_->WrappedObject();
[Current thread is 1 (Thread 0x7f9641872740 (LWP 30235))]
(gdb) bt
#0  0x0000000001a66b35 in node::MemoryRetainerNode::MemoryRetainerNode (this=0x6e95f10, tracker=0x7ffc9b58b9e0, retainer=0xb3be530) at ../src/memory_tracker-inl.h:29
#1  0x0000000001a67185 in node::MemoryTracker::AddNode (this=0x7ffc9b58b9e0, retainer=0xb3be530, edge_name=0x320201e "session") at ../src/memory_tracker-inl.h:263
#2  0x0000000001a6741a in node::MemoryTracker::PushNode (this=0x7ffc9b58b9e0, retainer=0xb3be530, edge_name=0x320201e "session") at ../src/memory_tracker-inl.h:297
#3  0x0000000001a66ff9 in node::MemoryTracker::Track (this=0x7ffc9b58b9e0, retainer=0xb3be530, edge_name=0x320201e "session") at ../src/memory_tracker-inl.h:244
#4  0x0000000001a66ea6 in node::MemoryTracker::TrackField (this=0x7ffc9b58b9e0, edge_name=0x320201e "session", value=0xb3be530, node_name=0x0) at ../src/memory_tracker-inl.h:100
#5  0x0000000001b455a6 in node::http2::Http2Session::Http2Settings::MemoryInfo (this=0xc3588f0, tracker=0x7ffc9b58b9e0) at ../src/node_http2.h:1160
#6  0x0000000001a6701b in node::MemoryTracker::Track (this=0x7ffc9b58b9e0, retainer=0xc3588f0, edge_name=0x0) at ../src/memory_tracker-inl.h:245
#7  0x0000000001a7bcb3 in node::Environment::<lambda(node::BaseObject*)>::operator()(node::BaseObject *) const (__closure=0x7ffc9b58b9d0, obj=0xc3588f0) at ../src/env.cc:718
#8  0x0000000001a7c0de in node::Environment::ForEachBaseObject<node::Environment::BuildEmbedderGraph(v8::Isolate*, v8::EmbedderGraph*, void*)::<lambda(node::BaseObject*)> >(<unknown type in /var/lib/nave/installed/10.16.3/bin/node, CU 0x2f408c, DIE 0x3762f7>) (this=0x7ffc9b591610, iterator=<unknown type in /var/lib/nave/installed/10.16.3/bin/node, CU 0x2f408c, DIE 0x3762f7>)
    at ../src/env-inl.h:856
#9  0x0000000001a7bd2c in node::Environment::BuildEmbedderGraph (isolate=0x64035e0, graph=0x7ffc9b58bbc0, data=0x7ffc9b591610) at ../src/env.cc:719
#10 0x0000000002859880 in v8::internal::HeapProfiler::BuildEmbedderGraph (this=0x6446730, isolate=0x64035e0, graph=0x7ffc9b58bbc0) at ../deps/v8/src/profiler/heap-profiler.cc:94
#11 0x00000000028773df in v8::internal::NativeObjectsExplorer::IterateAndExtractReferences (this=0x7ffc9b58bdf0, filler=0x7ffc9b58bc30) at ../deps/v8/src/profiler/heap-snapshot-generator.cc:2293
#12 0x0000000002878344 in v8::internal::HeapSnapshotGenerator::FillReferences (this=0x7ffc9b58bcf0) at ../deps/v8/src/profiler/heap-snapshot-generator.cc:2516
#13 0x00000000028780b9 in v8::internal::HeapSnapshotGenerator::GenerateSnapshot (this=0x7ffc9b58bcf0) at ../deps/v8/src/profiler/heap-snapshot-generator.cc:2470
#14 0x0000000002859998 in v8::internal::HeapProfiler::TakeSnapshot (this=0x6446730, control=0x0, resolver=0x0) at ../deps/v8/src/profiler/heap-profiler.cc:104
#15 0x0000000001e6a1c5 in v8::HeapProfiler::TakeHeapSnapshot (this=0x6446730, control=0x0, resolver=0x0) at ../deps/v8/src/api.cc:10480
#16 0x00007f9625b4d004 in (anonymous namespace)::WriteSnapshot(v8::FunctionCallbackInfo<v8::Value> const&) () from /var/cache/znpm64/heapdump-0.3.7/build/Release/addon.node
#17 0x0000000001f377d4 in v8::internal::FunctionCallbackArguments::Call (this=0x7ffc9b58d180, handler=0x1782bf8db941) at ../deps/v8/src/api-arguments-inl.h:94
#18 0x0000000001f3a5fa in v8::internal::(anonymous namespace)::HandleApiCallHelper<false> (isolate=0x64035e0, function=..., new_target=..., fun_data=..., receiver=..., args=...)
    at ../deps/v8/src/builtins/builtins-api.cc:109
#19 0x0000000001f384f0 in v8::internal::Builtin_Impl_HandleApiCall (args=..., isolate=0x64035e0) at ../deps/v8/src/builtins/builtins-api.cc:139
#20 0x0000000001f38271 in v8::internal::Builtin_HandleApiCall (args_length=6, args_object=0x7ffc9b58d3a0, isolate=0x64035e0) at ../deps/v8/src/builtins/builtins-api.cc:127
#21 0x000034e86e8c1d64 in ?? ()
#22 0x000034e86e8c1cc1 in ?? ()
#23 0x00007ffc9b58d350 in ?? ()
#24 0x0000000000000006 in ?? ()
#25 0x00007ffc9b58d3e8 in ?? ()
#26 0x000034e86e7a0ab6 in ?? ()
#27 0x00000ccf204026f1 in ?? ()
#28 0x00001782bf8db9d1 in ?? ()
#29 0x0000000600000000 in ?? ()
#30 0x00000ccf20402801 in ?? ()
#31 0x0000034306c9d369 in ?? ()
#32 0x00001e996428d481 in ?? ()
#33 0x00000ccf204022b1 in ?? ()
#34 0x00001782bf8db9d1 in ?? ()
#35 0x000039fb67984a19 in ?? ()
#36 0x0000034306c9d3a1 in ?? ()
#37 0x0000005000000000 in ?? ()
#38 0x000007754167fe01 in ?? ()
#39 0x00001782bf8dba79 in ?? ()
#40 0x00002b96738db5e1 in ?? ()
#41 0x00007ffc9b58d420 in ?? ()
#42 0x000034e86e78f303 in ?? ()
#43 0x0000088e1c58a771 in ?? ()
#44 0x0000000000000000 in ?? ()

in node source code added 2 lines - see #29902 (comment)

[gireeshpunathil]

A wild guess is that the retainer pointer is garbage. 0xb3be530 does not look like a valid native address, as it is too distant from the other ones used - 0x7ffcxxxxxxxx. Can you pls also dump the registers (info registers) and few instructions before the RIP? (x/10i 0x1a66b00)

[lomaster1]

It's not the same as above but maybe helpful to you.

GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /var/lib/nave/installed/10.16.3/bin/node...done.
[New LWP 26735]
[New LWP 26828]
[New LWP 26826]
[New LWP 26831]
[New LWP 26825]
[New LWP 26742]
[New LWP 26747]
[New LWP 26737]
[New LWP 26746]
[New LWP 26739]
[New LWP 26744]
[New LWP 26740]
[New LWP 26830]
[New LWP 26829]
[New LWP 26827]
[New LWP 26824]
[New LWP 26745]
[New LWP 26741]
[New LWP 26738]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/var/lib/nave/installed/10.16.3/bin/node --expose-internals --expose-gc --max-h'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000000000095ff99 in node::MemoryTracker::Track(node::MemoryRetainer const*, char const*) ()
[Current thread is 1 (Thread 0x7fd715e2f740 (LWP 26735))]
(gdb) bt
#0  0x000000000095ff99 in node::MemoryTracker::Track(node::MemoryRetainer const*, char const*) ()
#1  0x00000000008e3c2a in node::Environment::BuildEmbedderGraph(v8::Isolate*, v8::EmbedderGraph*, void*) ()
#2  0x00000000010ca894 in v8::internal::HeapProfiler::BuildEmbedderGraph(v8::internal::Isolate*, v8::EmbedderGraph*) ()
#3  0x00000000010dfae2 in v8::internal::NativeObjectsExplorer::IterateAndExtractReferences(v8::internal::SnapshotFiller*) ()
#4  0x00000000010e0e21 in v8::internal::HeapSnapshotGenerator::GenerateSnapshot() ()
#5  0x00000000010cb634 in v8::internal::HeapProfiler::TakeSnapshot(v8::ActivityControl*, v8::HeapProfiler::ObjectNameResolver*) ()
#6  0x00007fd6fdcfe000 in (anonymous namespace)::WriteSnapshot(Nan::FunctionCallbackInfo<v8::Value> const&) () from /var/cache/znpm64/heapdump-0.3.15/build/Release/addon.node
#7  0x00007fd6fdcfded7 in Nan::imp::FunctionCallbackWrapper(v8::FunctionCallbackInfo<v8::Value> const&) () from /var/cache/znpm64/heapdump-0.3.15/build/Release/addon.node
#8  0x0000000000b8e6af in v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) ()
#9  0x0000000000b8f219 in v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) ()
#10 0x000023936ed5be1d in ?? ()
#11 0x0000271ad4471939 in ?? ()
#12 0x000023936ed5bd81 in ?? ()
#13 0x00007ffde0a4c010 in ?? ()
#14 0x0000000000000006 in ?? ()
#15 0x00007ffde0a4c0c0 in ?? ()
#16 0x000023936ed118d5 in ?? ()
#17 0x000027b62cf826f1 in ?? ()
#18 0x000010e8bb748d89 in ?? ()
#19 0x0000000600000000 in ?? ()
#20 0x000027b62cf82801 in ?? ()
#21 0x000027b62cf826f1 in ?? ()
#22 0x000013aa4d8ce3d9 in ?? ()
#23 0x000027b62cf826f1 in ?? ()
#24 0x000027b62cf826f1 in ?? ()
#25 0x000013aa4d8ce3d9 in ?? ()
#26 0x000010e8bb748d89 in ?? ()
#27 0x000027b62cf826f1 in ?? ()
#28 0x000027b62cf826f1 in ?? ()
#29 0x0000005000000000 in ?? ()
#30 0x00002cd2c6c5f1e1 in ?? ()
#31 0x000010e8bb748fe1 in ?? ()
#32 0x00000e1e9077fe49 in ?? ()
#33 0x00007ffde0a4c108 in ?? ()
#34 0x000023936ed0a5c3 in ?? ()
#35 0x00002d08555fdc31 in ?? ()
#36 0x000027b62cf826f1 in ?? ()
#37 0x000016b5be930029 in ?? ()
#38 0x0000000000000000 in ?? ()
(gdb) info registers
rax            0x0	0
rbx            0x4b747d0	79120336
rcx            0x7fd714f662c0	140561746387648
rdx            0x1	1
rsi            0x1	1
rdi            0x4b747d0	79120336
rbp            0x7ffde0a4a370	0x7ffde0a4a370
rsp            0x7ffde0a4a2f0	0x7ffde0a4a2f0
r8             0x7fd714d48bc8	140561744169928
r9             0x5e12180	98640256
r10            0x7d	125
r11            0x7ffde0a4a3f0	140728372339696
r12            0x7ffde0a4a3f0	140728372339696
r13            0xc5ba250	207331920
r14            0x7ffde0a4a300	140728372339456
r15            0x7ffde0a4a710	140728372340496
rip            0x95ff99	0x95ff99 <node::MemoryTracker::Track(node::MemoryRetainer const*, char const*)+233>
eflags         0x10202	[ IF RF ]
cs             0x33	51
ss             0x2b	43
ds             0x0	0
es             0x0	0
fs             0x0	0
gs             0x0	0
k0             0x0	0
k1             0x0	0
k2             0x0	0
k3             0x0	0
k4             0x0	0
k5             0x0	0
k6             0x0	0
k7             0x0	0
(gdb) x/10i 0x000000000095ff99
=> 0x95ff99 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+233>:	callq  *0x28(%rax)
   0x95ff9c <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+236>:	test   %rax,%rax
   0x95ff9f <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+239>:	je     0x95ffbd <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+269>
   0x95ffa1 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+241>:	mov    0x8(%r12),%rdi
   0x95ffa6 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+246>:	lea    -0x70(%rbp),%r14
   0x95ffaa <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+250>:	mov    %r14,%rsi
   0x95ffad <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+253>:	mov    (%rdi),%rdx
   0x95ffb0 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+256>:	mov    (%rdx),%rdx
   0x95ffb3 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+259>:	mov    %rax,-0x70(%rbp)
   0x95ffb7 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+263>:	callq  *%rdx

I won't delete this core dump. So you can ask me for additional info

[gireeshpunathil]

few instructions before the RIP pls? (x/10i 0x000000000095ff80)

[bnoordhuis]

And can you post the output of info files?

[lomaster1]

x/10i 0x000000000095ff80

(gdb) x/10i 0x000000000095ff80
   0x95ff80 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+208>:	add    %cl,-0x75(%rcx)
   0x95ff83 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+211>:	xor    $0x24,%al
   0x95ff85 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+213>:	lea    -0x40(%rbp),%rdi
   0x95ff89 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+217>:	lea    -0x70(%rbp),%r14
   0x95ff8d <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+221>:	callq  0xb012b0 <_ZN2v811HandleScopeC2EPNS_7IsolateE>
   0x95ff92 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+226>:	mov    0x8(%r13),%rdi
   0x95ff96 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+230>:	mov    (%rdi),%rax
=> 0x95ff99 <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+233>:	callq  *0x28(%rax)
   0x95ff9c <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+236>:	test   %rax,%rax
   0x95ff9f <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+239>:	je     0x95ffbd <_ZN4node13MemoryTracker5TrackEPKNS_14MemoryRetainerEPKc+269>

[lomaster1]

And can you post the output of info files?

(gdb) info files
Symbols from "/var/lib/nave/installed/10.16.3/bin/node".
Local core dump file:
	`/var/crash/1/./CoreDump', file type elf64-x86-64.
	0x0000000000400000 - 0x0000000000401000 is load1a
	0x0000000000401000 - 0x0000000000401000 is load1b
	0x0000000002645000 - 0x0000000002647000 is load2
	0x0000000002647000 - 0x0000000002654000 is load3
	0x0000000002654000 - 0x000000000266d000 is load4
	0x00000000034c7000 - 0x000000001e090000 is load5
	0x0000002b14200000 - 0x0000002b14280000 is load6
	0x0000002e54c00000 - 0x0000002e54c80000 is load7
	0x0000003ef1c00000 - 0x0000003ef1c80000 is load8
	0x0000006d8ff00000 - 0x0000006d8ff80000 is load9
	0x000000907f380000 - 0x000000907f44c000 is load10
	0x00000095a1480000 - 0x00000095a1500000 is load11
	0x000000b67d280000 - 0x000000b67d300000 is load12
	0x000000f090200000 - 0x000000f090280000 is load13
	0x0000011d7ee80000 - 0x0000011d7ef00000 is load14
	0x0000014dfa200000 - 0x0000014dfa280000 is load15
	0x0000015956080000 - 0x0000015956100000 is load16
	0x000001619b800000 - 0x000001619b880000 is load17
	0x0000016815f00000 - 0x0000016815f80000 is load18
	0x00000198f6a00000 - 0x00000198f6a80000 is load19
	0x000001a3c7e00000 - 0x000001a3c7e80000 is load20
	0x000001c3bb780000 - 0x000001c3bb800000 is load21
	0x000001c97d700000 - 0x000001c97d780000 is load22
	0x000001ca8d200000 - 0x000001ca8d280000 is load23
	0x000001e4feb80000 - 0x000001e4fec00000 is load24
	0x000001f2ef100000 - 0x000001f2ef180000 is load25
	0x000001fa74080000 - 0x000001fa74100000 is load26
	0x0000020de5380000 - 0x0000020de5383000 is load27
	0x000002233c080000 - 0x000002233c100000 is load28
	0x000002266e180000 - 0x000002266e200000 is load29
	0x000002303a500000 - 0x000002303a580000 is load30
	0x0000025472700000 - 0x0000025472780000 is load31
	0x00000268be380000 - 0x00000268be400000 is load32
	0x0000026976580000 - 0x0000026976600000 is load33
	0x000002875ea00000 - 0x000002875ea39000 is load34
	0x000002ac9e1e6000 - 0x000002ac9e1e6000 is load35
	0x000002ac9e200000 - 0x000002ac9e203000 is load36
	0x000002ac9e203000 - 0x000002ac9e203000 is load37
	0x000002ac9e204000 - 0x000002ac9e27f000 is load38
	0x000002ac9e27f000 - 0x000002ac9e27f000 is load39
	0x000002ac9e280000 - 0x000002ac9e283000 is load40
	0x000002ac9e283000 - 0x000002ac9e283000 is load41
	0x000002ac9e284000 - 0x000002ac9e2ff000 is load42
	0x000002ac9e2ff000 - 0x000002ac9e2ff000 is load43
	0x000002ac9e300000 - 0x000002ac9e303000 is load44
	0x000002ac9e303000 - 0x000002ac9e303000 is load45
	0x000002ac9e304000 - 0x000002ac9e319000 is load46

[bnoordhuis]

Thanks. Do you have any other add-ons besides node-heapdump loaded? What does find node_modules -name \*.node print?

[lomaster1]

Yes, I have.
I send to you not a full list of files. Here is full files.txt

[bnoordhuis]

Right, I can see you're using a bunch of add-ons. Odds are > 95% the crash is caused by one of them; ffi (and ref, and ref-struct, etc.) in particular are great at corrupting the JS heap. Try excluding them and see if the crash goes away.

If you can still reproduce without add-ons (heapdump excepted), please post steps to reproduce and I'll take a look.