received SIGSEGV for address: 0x1aa00030f3e , Many times

[magicode]

in node v4.2.4

[Tue Dec 29 2015 20:54:58 GMT+0000]  [ code exit 255 ]
PID 32450 received SIGSEGV for address: 0x1aa00030f3e
/home/app/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x1af4)[0x7f7480a9caf4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f7483a7d340]
app (_ZN2v88internal18IncrementalMarking4StepElNS1_16CompletionActionENS1_18ForceMarkingActionENS1_21ForceCompletionActionE+0x17)[0xadf0e7]
app (_ZN2v88internal8NewSpace15SlowAllocateRawEiNS0_19AllocationAlignmentE+0x74)[0xb0eb94]
app (_ZN2v88internal4Heap11AllocateRawEiNS0_15AllocationSpaceES2_NS0_19AllocationAlignmentE+0x1b9)[0xa70ae9]
app (_ZN2v88internal4Heap8AllocateEPNS0_3MapENS0_15AllocationSpaceEPNS0_14AllocationSiteE+0x41)[0xabe701]
app (_ZN2v88internal4Heap16AllocateJSObjectEPNS0_10JSFunctionENS0_13PretenureFlagEPNS0_14AllocationSiteE+0x59)[0xac0599]
app (_ZN2v88internal7Factory11NewJSObjectENS0_6HandleINS0_10JSFunctionEEENS0_13PretenureFlagE+0x37)[0xa799c7]
app [0xcb13cd]
app (_ZN2v88internal17Runtime_NewObjectEiPPNS0_6ObjectEPNS0_7IsolateE+0x31)[0xcb7871]

[Tue Dec 29 2015 20:55:03 GMT+0000] [ code exit 255 ]
PID 26868 received SIGSEGV for address: 0x1ae00036ed6
/home/app/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x1af4)[0x7f0391533af4]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f0394514340]
app (_ZN2v88internal18IncrementalMarking21RecordCodeTargetPatchEPhPNS0_10HeapObjectE+0x1f)[0xadd63f]
app (_ZN2v88internal2IC10set_targetEPNS0_4CodeE+0xaa)[0xb9630a]
app (_ZN2v88internal2IC19UpdatePolymorphicICENS0_6HandleINS0_4NameEEENS2_INS0_4CodeEEE+0x415)[0xb976f5]
app (_ZN2v88internal2IC10PatchCacheENS0_6HandleINS0_4NameEEENS2_INS0_4CodeEEE+0x63)[0xb97953]
app (_ZN2v88internal7StoreIC12UpdateCachesEPNS0_14LookupIteratorENS0_6HandleINS0_6ObjectEEENS5_14StoreFromKeyedE+0x17e)[0xb9b52e]
app (_ZN2v88internal7StoreIC5StoreENS0_6HandleINS0_6ObjectEEENS2_INS0_4NameEEES4_NS3_14StoreFromKeyedE+0x455)[0xb9ba35]
app (_ZN2v88internal12StoreIC_MissEiPPNS0_6ObjectEPNS0_7IsolateE+0x192)[0xb9ca22]
[0x2db12d20963b]

[mscdex]

Just to be extra sure about the stack trace, can you run without segfault-handler and use something like gdb or another debugger instead?

[mscdex]

Also, what are you doing to trigger this? Are you merely starting node at the shell prompt? Are you running a particular script? Something else?

What platform is this on? Are you using third party addons?

[magicode]

this proxy server.
according to the log it seems errors that occur in idle
in node 0.10 or 0.12 does not happen

i use addons.
https://github.com/magicode/img-netfree
https://github.com/magicode/mmmagic
https://github.com/magicode/node-mtrace
https://github.com/magicode/nat-lookup
https://github.com/magicode/openssl-ca-node
https://github.com/magicode/remove-skin
segfault-handler

[magicode]

i check log
error at

var tlssocket = new tls.TLSSocket(socket , { secureContext: credential, isServer:  true });

[mscdex]

Do you have a simple, reproducible example that does not include any third party modules or addons? Also, what platform is this on (e.g. Windows, Linux, OS X, FreeBSD, etc.)?

[magicode]

I try to create a simple reproducible example. So far unsuccessfully.
platform is linux ubuntu 14

[magicode]

Also, Do you have an idea how I can debug it automatically in production

[Yaakov-Belch]

Can you replay a request sequence (copied from production) against a non-production proxy?
You may get the sequence either from your log files or capture them from a live production run.

This should produce a reproduction of this bug that can be worked on outside of production.

[ofrobots]

For readability, here's the demangled stack trace 1:

app (v8::internal::IncrementalMarking::Step(long, v8::internal::IncrementalMarking::CompletionAction, v8::internal::IncrementalMarking::ForceMarkingAction, v8::internal::IncrementalMarking::ForceCompletionAction) 0x17)[0xadf0e7]
app (v8::internal::NewSpace::SlowAllocateRaw(int, v8::internal::AllocationAlignment) 0x74)[0xb0eb94]
app (v8::internal::Heap::AllocateRaw(int, v8::internal::AllocationSpace, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) 0x1b9)[0xa70ae9]
app (v8::internal::Heap::Allocate(v8::internal::Map*, v8::internal::AllocationSpace, v8::internal::AllocationSite*) 0x41)[0xabe701]
app (v8::internal::Heap::AllocateJSObject(v8::internal::JSFunction*, v8::internal::PretenureFlag, v8::internal::AllocationSite*) 0x59)[0xac0599]
app (v8::internal::Factory::NewJSObject(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::PretenureFlag) 0x37)[0xa799c7]
app [0xcb13cd]
app (v8::internal::Runtime_NewObject(int, v8::internal::Object**, v8::internal::Isolate*) 0x31)[0xcb7871]

Stack trace 2:

app (v8::internal::IncrementalMarking::RecordCodeTargetPatch(unsigned char*, v8::internal::HeapObject*) 0x1f)[0xadd63f]
app (v8::internal::IC::set_target(v8::internal::Code*) 0xaa)[0xb9630a]
app (v8::internal::IC::UpdatePolymorphicIC(v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Code>) 0x415)[0xb976f5]
app (v8::internal::IC::PatchCache(v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Code>) 0x63)[0xb97953]
app (v8::internal::StoreIC::UpdateCaches(v8::internal::LookupIterator*, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) 0x17e)[0xb9b52e]
app (v8::internal::StoreIC::Store(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Name>, v8::internal::Handle<v8::internal::Object>, v8::internal::Object::StoreFromKeyed) 0x455)[0xb9ba35]
app (v8::internal::StoreIC_Miss(int, v8::internal::Object**, v8::internal::Isolate*) 0x192)[0xb9ca22]

[Yaakov-Belch]

It seems that both exceptions happened during garbage collection --- but in different parts of the program execution.

Quite likely, some part of the program corrupts memory, including control structures used for garbage collection. This throws an exception whenever the affected piece of memory is collected.

Stack trace 1 could quite possibly show how creating a new object (e.g. new tls.TLSSocket(...)) requires allocating new free memory which in turn triggers an incremental garbage collection and triggers the exception.

In Stack trace 2, garbage collection is triggered by another function.

[ofrobots]

Assuming the stack trace is from an x86-64 machine, In both cases we crash because heap::incremental_marking_ looks like an invalid pointer. This does smell like memory corruption, but it is too coincidental that the same pointer is getting corrupted in both cases.

It would be to good to have a test-case reproducing this issue without third-party native addons.

(btw, I wouldn't characterize the second stack trace as 'garbage collection is triggered'. Rather we are doing book-keeping for GC to use later).

[ghost]

@magicode @mscdex @ofrobots @Trott @Yaakov-Belch Execute me, What's this going on? I updated my addon to support nan 2.2.0, node 4.4.1 , got same error. Is there a way to solve this?

https://github.com/zwb-ict/lmdb-queue

[mscdex]

I wonder if this may be the same/related issue as #5900?

[ofrobots]

Based on the stack trace from before, this doesn't look like a duplicate of #5900. @zwb-ict Can you include the exact stack trace you see?

[magicode]

in node 6, this bug solved

[jasnell]

Awesome to hear... if we can track down what fixed it perhaps we can backport ;-)

[bnoordhuis]

Closing, inactive for > 6 months. If someone knows of a reliable way to reproduce, let me know and I'll reopen.

[CExAdamJ]

I believe I'm getting the same issue with Node v4.8.4. Any chance of this being backported? I would really appreciate a simple patch w/o having to upgrade older services to v6.

PID 8 received SIGSEGV for address: 0xbd673a01204
/isight/node_modules/isight/node_modules/segfault-handler/build/Release/segfault-handler.node(+0x1a5b)[0x7fcdb2b14a5b]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xf890)[0x7fcdb50d4890]
node(_ZN2v88internal20MarkCompactCollector22ProcessWeakCollectionsEv+0xf1)[0xaed6c1]
node(_ZN2v88internal20MarkCompactCollector15MarkLiveObjectsEv+0x1a4)[0xaf5a24]
node(_ZN2v88internal20MarkCompactCollector14CollectGarbageEv+0x11)[0xaf64c1]
node(_ZN2v88internal4Heap11MarkCompactEv+0x60)[0xaad640]
node(_ZN2v88internal4Heap24PerformGarbageCollectionENS0_16GarbageCollectorENS_15GCCallbackFlagsE+0x318)[0xac5068]
node(_ZN2v88internal4Heap14CollectGarbageENS0_16GarbageCollectorEPKcS4_NS_15GCCallbackFlagsE+0x239)[0xac5609]
node(_ZN2v88internal4Heap15HandleGCRequestEv+0xa1)[0xac6011]
node(_ZN2v88internal10StackGuard16HandleInterruptsEv+0x31c)[0xa61c6c]
node(_ZN2v88internal18Runtime_StackGuardEiPPNS0_6ObjectEPNS0_7IsolateE+0x2b)[0xc978db]
[0x2f841fd060bb]

[ofrobots]

@CExAdamJ we would need a test case to reproduce the issue (that doesn't have any native module dependencies) to be able to make progress on this.