configuration option --gdb not working properly

[trevnorris]

After running $ ./configure --gdb; make it should be possible to do this from gdb:

$ gdb --args ./iojs --gdbjit /tmp/crash.js
(gdb) break __jit_debug_register_code

And it should break like so:

Starting program: /var/projects/nubjs-deps/v8/out/x64.release/d8 --gdbjit /tmp/crash.js
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff53b4700 (LWP 8650)]
[New Thread 0x7ffff5bb5700 (LWP 8649)]
[New Thread 0x7ffff63b6700 (LWP 8648)]
[New Thread 0x7ffff6bb7700 (LWP 8647)]

Breakpoint 1, 0x00000000004dbc00 in __jit_debug_register_code ()

But it doesn't. The above was captured from d8 setting ENABLE_GDB_JIT_INTERFACE.

I'll look into this later. Just filing a proper issue.

EDIT: Missing breakpoint is only exhibited on current next branch.

[trevnorris]

Running the same script with current master I get the following stack trace:

$ gdb --args ./iojs --gdbjit --gdbjit-full /tmp/crash.js
...
(gdb) bt
#0  0x00007ffff6bed267 in __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:55
#1  0x00007ffff6beeeca in __GI_abort () at abort.c:89
#2  0x0000000000c0e304 in node::OnFatalError(char const*, char const*) ()
#3  0x00000000007a9b12 in v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) ()
#4  0x00000000008f1490 in v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) ()
#5  0x0000000000afa62e in v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) ()
#6  0x00001f5e12f062b5 in ?? ()
#7  0x3030393334382e30 in ?? ()
#8  0x1c7b3b0900003a91 in ?? ()
#9  0x7b3400000032550e in ?? ()

So even though the breakpoint is hit on master, it doesn't give us any information. Though d8 on the head of 4.2 branch gives:

$ gdb --args ./out/x64.release/d8 --gdbjit --gdbjit-full /tmp/crash.js
...
(gdb) bt
#0  0x000000000086c16f in v8::base::OS::Abort() ()
#1  0x000000000040f931 in v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) ()
#2  0x00000000004b41ed in v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) ()
#3  0x00000000007d623b in v8::internal::Runtime_AllocateInTargetSpace(int, v8::internal::Object**, v8::internal::Isolate*) ()
#4  0x00000eb7bf807d95 in Stub:CEntryStub ()
#5  0x00000eb7bf90ac53 in Stub:StringAddStub_Tenured ()
#6  0x00000eb7bf9077d2 in zzz () at /tmp/crash.js:1
#7  0x00000eb7bf858cae in yyy () at /tmp/crash.js:3
#8  0x00000eb7bf858dae in xxx () at /tmp/crash.js:2
#9  0x00000eb7bf858861 in /tmp/crash.js () at /tmp/crash.js:15

Not sure how this could be happening.

[trevnorris]

@bnoordhuis Sorry to pull you in on this one, but cannot figure out how a build of iojs with --gdb would show no output, but a build of same patch version of d8 with it enabled would show the JS output. Any thoughts where I could dig to find the answer?

EDIT: Interesting note that d8 shows similar output to iojs if the --gdbjit-full is removed. Wonder if additional information isn't being emitted to gdb.

[ofrobots]

@trevnorris I'll try to reproduce. In the meanwhile there is another trick you can try. After you have hit the out-of-memory abort, run this in gdb:

(gdb) source $path_to_node/deps/v8/tools/gdbinit
(gdb) jst

This should give you the javascript stacktrace (with a lot more verbose information). Note that you don't really need to reconfigure with --gdb for this to work.

[ofrobots]

@trevnorris Do you know the last time --gdbjit and --gdbjit-full worked for you? I can go all the way back to 1.0.0 and it still doesn't work. Whatever is broken, hasn't broken recently.

Looking at the V8 code in gdb-jit.cc, I don't see anything obviously wrong, and as you mention d8 seems to work okay. Also, I have verified, using --gdbjit-dump that the object files being produced by gdb-jit.cc have decent symbols.

My suspicion is that there is something different about memory layout/map for the node process that confuses gdb somehow.

[ofrobots]

Removed the build tag as the --gdb configure option does correctly enable gdb support in V8.

[bnoordhuis]

I'm not 100% sure but I believe gdbjit was broken until 4.3. The commit that fixes it (if I'm right) is v8/v8@652061e and is in next.

[coreyfarrell]

Maybe worth installing gdbinit to $PREFIX/share/io.js, documenting it's use? It's not impossible to crash iojs / nodejs so this seems useful, especially if it can be used without recompiling iojs.

[ofrobots]

@coreyfarrell 👍. It is much easier to use gdbinit than recompiling with --gdb. Jit support in gdb happens to be buggy and slow anyway. PRs are welcome, or I will try to do it once I get a chance.

[ofrobots]

@bnoordhuis gdbjit still doesn't seem to work for me with node in next, or any release of io.js (that I tried) going back to 1.0. Note that it does work with d8 in atleast 4.2. I think this is something node-specific, but I am not entirely sure what yet.

[trevnorris]

@ofrobots The issue with current master and next are different. In master I can hit the __jit_debug_register_code breakpoint, but no bt output. In next branch the breakpoint is never reached.

node v0.11.9 would show the first JS point in the backtrace output with gdb. Back at that time I would use lldb because the output was more comprehensive. Then lldb for linux x64 got borked and I stopped using the feature.

Recently gave it a shot against d8 and found that it mostly worked. Though both options must be passed. But even with both options passed to io.js the end of the stack looks like so:

#8  0x000027ddf1006001 in ?? ()
#9  0x00007fffffffd530 in ?? ()
#10 0x00007fffffffd598 in ?? ()
#11 0x000027ddf10e864e in zzz ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

[ofrobots]

I think this is a gdb-related issue. This is how far I got:

V8 is reporting the code-compile events to gdb by calling __jit_debug_register_code. If you enable internal gdb debugging using the set debug jit 1 command, you can see that gdb acknowledges the receipt of these symbols. However, for some reason it doesn't use them during the backtrace. I am not sure why this works with d8 but not node.

lemongrab :: ~/src/io.js ‹master› % gdb --args ./iojs --gdbjit --gdbjit-full /tmp/crash.js
...
(gdb) set debug jit 10
(gdb) r
Starting program: /ofrobots/src/io.js/out/Release/iojs --gdbjit --gdbjit-full /tmp/crash.js
jit_breakpoint_re_set_internal, breakpoint_addr = 0x9b9a30
jit_breakpoint_re_set_internal, breakpoint_addr = 0x9b9a30
warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000
jit_breakpoint_re_set_internal, breakpoint_addr = 0x9b9a30
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
jit_breakpoint_re_set_internal, breakpoint_addr = 0x9b9a30
[New Thread 0x7ffff6bca700 (LWP 7994)]
[New Thread 0x7ffff63c9700 (LWP 7995)]
[New Thread 0x7ffff5bc8700 (LWP 7996)]
[New Thread 0x7ffff53c7700 (LWP 7997)]
[New Thread 0x7ffff4bc6700 (LWP 7998)]
jit_read_descriptor, descriptor_addr = 0x1343cc0
jit_register_code, symfile_addr = 0x13a2fe0, symfile_size = 688
jit_register_code, symfile_addr = 0x13a2fe0, symfile_size = 688
jit_breakpoint_re_set_internal, breakpoint_addr = 0x9b9a30
jit_read_descriptor, descriptor_addr = 0x1343cc0
jit_register_code, symfile_addr = 0x139e490, symfile_size = 699
jit_register_code, symfile_addr = 0x139e490, symfile_size = 699
jit_breakpoint_re_set_internal, breakpoint_addr = 0x9b9a30
jit_read_descriptor, descriptor_addr = 0x1343cc0
...

The above was with io.js @ master. To compare I used a d8 from branch-heads/4.2.

[Fishrock123]

Is this still not working correctly?

[trevnorris]

from @bnoordhuis' comment I believe it won't work properly until we bring in 4.7.3+.

[ofrobots]

@trevnorris where are you getting the 4.7.3 from? The commit @bnoordhuis referenced landed in 4.3, and has been available in node v3.x and master.

Last time I tried, it did not fix the issue for me though. It works fine with d8, but not with node.

I personally think that gdbjit is not useful because of the limitations and gdb bugs. I find that the gdbinit (#2123) works better in more scenarios without requiring a rebuild, and without adversely affecting performance.

Are there use-cases where people still want to use --gdbjit?

[trevnorris]

@ofrobots my bad. I misread the tag below the commit message on GitHub.

[trevnorris]

As for gdbjit, IIRC it was useful to be able to step through the JS source in gdb. gdbinit definitely gives more information, but being able to step through the JS itself in gdb was also useful.

[ofrobots]

Okay, that is a reasonable use-case. I don't have bandwidth at the moment, but I will try to take another look when I get some spare cycles. Others are welcome to investigate this too.

[ofrobots]

Self-assigning so that I don't forget, but others are welcome to investigate it too.

[trevnorris]

Thanks. Seems it has to be something like build configuration. Since it works perfectly in d8. I'll also try to take a peek there.

[Trott]

Any chance someone knows if this is resolved or still an issue?

[trevnorris]

@Trott This unfortunately is still an issue. Building [email protected] (same as current master) and running the simple script:

performance.now();

and executing with:

> lldb-3.6 -- ./out/x64.release/d8 --gdbjit --gdbjit-full script.js

we can see a significant amount of information. For example:

  * frame #0: 0x0000000000406300 d8`v8::Shell::PerformanceNow(v8::FunctionCallbackInfo<v8::Value> const&)
    frame #1: 0x0000000000838719 d8`v8::internal::FunctionCallbackArguments::Call(void (*)(v8::FunctionCallbackInfo<v8::Value> const&)) + 153
    frame #2: 0x00000000004613d0 d8`v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::(anonymous namespace)::BuiltinArguments<(v8::internal::BuiltinExtraArguments)1>) + 960
    frame #3: 0x000000000046f654 d8`v8::internal::Builtin_HandleApiCall(int, v8::internal::Object**, v8::internal::Isolate*) + 52
    frame #4: 0x0000118c6f60c17b JIT(0xbc3f30)`Stub:CEntryStub + 91
    frame #5: 0x0000118c6f63cf93 JIT(0xc31050)`Script:script.js + 115 at script.js:3
    frame #6: 0x0000118c6f636904 JIT(0xc1feb0)`Builtin:JSEntryTrampoline + 132
(lldb) f 5
frame #5: 0x0000118c6f63cf93 JIT(0xc31050)`Script:script.js + 115 at script.js:1
-> 1    performance.now();
(lldb) di -f
   0x118c6f63cf20 <Script:script.js>: pushq  %rbp
   0x118c6f63cf21 <Script:script.js+1>: movq   %rsp, %rbp
   0x118c6f63cf24 <Script:script.js+4>: pushq  %rsi
   0x118c6f63cf25 <Script:script.js+5>: pushq  %rdi
   0x118c6f63cf26 <Script:script.js+6>: pushq  -0x58(%r13)
   0x118c6f63cf2a <Script:script.js+10>: cmpq   0xa78(%r13), %rsp
   0x118c6f63cf31 <Script:script.js+17>: jae    0x118c6f63cf38            ; Script:script.js + 24 at script.js:1
   0x118c6f63cf33 <Script:script.js+19>: callq  0x118c6f6380a0            ; Builtin:StackCheck
...

This is excellent we can walk through the JS on a release build.

Unfortunately on master building with ./configure --gdb --debug and adding CXXFLAGS="-g" we still can't get any information even running the debug build. I've attempted to fix this several times, since this functionality could greatly improve certain debugging cases, but haven't figured out what's going on. I'd love if someone else could pick this up and figure out what's wrong.

[trevnorris]

@bnoordhuis Awesome. Thanks much for taking care of this.