Invalid memory access (signal 11) at address 0x48

[codenoid]

I dont know where it happened, but obviously my application has been running for 4 days +,
HTTP::Client, html parser, db connection (mongocr), etc
Spec

$ Crystal 0.23.1 [e2a1389] (2017-07-13) LLVM 3.8.1
$ Ubuntu 16.04.3 64bit

Log :

Invalid memory access (signal 11) at address 0x48
[0x454def] __crystal_sigfault_handler +63
[0x7f9c9c8f5390] ???
[0x434ae4] ???
[0x45475f] main +47
[0x7f9c9bf18830] __libc_start_main +240
[0x42f199] _start +41
[0x0] ???

note : i dont know how to reproduce this bug.

[ysbaddaden]

We can't reproduce without some reduced code sample. The backtrace doesn't tell anything :(

[codenoid]

@ysbaddaden my code only contains looping and http client, I dont know which one is wrong, because when 4 days it's run without a problem :/

def list() # dynamic data
  a = [] of String
  db = data_from_mongo
  db.each do |v|
    a << v
  end
  return a
end

loop do
  list().each do |v|
    # some HTTP CLIENT
    # HTML PARSER (modest and myhtml)
    # json parser,
    # mongo insert and update method
  end
end

[RX14]

The best way to debug this is to disable crystals dig fault hander and turn on coredump collection. With that and the built executable we could debug this.

Unfortunately turning off the sigfault handler requires editing the stdlib currently.

[bararchy]

@RX14 would be nice to have that auto disabled in debug mode or something , or at least --coredump (w.e makes sense)

[RX14]

The sigfault handler is part of the stdlib so it should be a flag using -D no_sigfault_handler

[t-richards]

For what it's worth, it looks like the sigfault handler can be disabled without editing the stdlib (please correct me if I'm wrong!).

In the following example, I'm running a poorly-written program which uses some unsafe stuff incorrectly (hey, that backtrace looks very similar...)

$ crystal run src/main.cr 
Invalid memory access (signal 11) at address 0x48
[0x55751dc571e6] *CallStack::print_backtrace:Int32 +118
[0x55751dc4bc6d] __crystal_sigfault_handler +61
[0x7fc1c50d9da0] ???
[0x55751dc953c4] *TCPSocket+ +4
[0x55751dc3c473] ???
[0x55751dc4bb69] main +41
[0x7fc1c44b8f6a] __libc_start_main +234
[0x55751dc3bcaa] _start +42
[0x0] ???

I added the following lines to the top of my program:

👇 👇 👇

Signal::SEGV.reset
Signal::BUS.reset

☝️ ☝️ ☝️

This allowed it to receive the SEGV.

$ crystal run src/main.cr 
Program exited because of a segmentation fault (11)

Now, I can run it through GDB to find the source of the issue:

$ crystal build src/main.cr
$ gdb ./main
(gdb) run

Thread 1 "main" received signal SIGSEGV, Segmentation fault.
0x00005555555e53f4 in close () at /usr/lib/crystal/io/buffered.cr:194
194	    flush if @out_count > 0
(gdb) where
#0  0x00005555555e53f4 in close () at /usr/lib/crystal/io/buffered.cr:194
#1  0x000055555558c49e in __crystal_main () at /tmp/cruddy-program/src/main.cr:19
#2  0x000055555559bb99 in main () at /usr/lib/crystal/main.cr:12

Even though it looks like there was an issue in the IO library, the real problem was that I used some uninitialized memory incorrectly. Attempting to access the garbage address of @out_count caused a segmentation fault.

Hope this helps! 😄

[RX14]

gdb should break on signals before the crystal fault handler runs. I never had to disable the sigfault handler when using gdb, but it does prevent enabling coredumps outside gdb. Useful technique though.

[asynkayo]

Hi, I found a similar issue I guess (Crystal 0.23.1 (2017-09-10) LLVM 5.0.0 on linux 64bits):

Code:
puts "Hello"

Build:
crystal build -d --release file.cr

Output:
Invalid memory access (signal 11) at address 0x1c
[0x5636b112df2f] __crystal_sigfault_handler +63
[0x7faabcca77e0] ???
[0x7faabe8b0f80] ???
[0x7faabe8ccb0d] _ZN4llvm10DwarfDebug25finishVariableDefinitionsEv +45
[0x7faabe8db626] _ZN4llvm10DwarfDebug18finalizeModuleInfoEv +70
[0x7faabe8dba5a] _ZN4llvm10DwarfDebug9endModuleEv +26
[0x7faabe8a5b8f] _ZN4llvm10AsmPrinter14doFinalizationERNS_6ModuleE +591
[0x7faabe30edb9] _ZN4llvm13FPPassManager14doFinalizationERNS_6ModuleE +73
[0x7faabe31b145] _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE +885
[0x7faabf4bd73e] ???
[0x7faabf4bdbe4] LLVMTargetMachineEmitToFile +244
[0x5636b18b233a] ???
[0x5636b1825aae] ???
[0x5636b1b7c1f5] ???
[0x5636b11578c6] ???
[0x5636b112d6dc] main +13628
[0x7faabc0974ca] __libc_start_main +234
[0x5636b11067ba] _start +42
[0x0] ???

[Sija]

@Booss duplicate of #4570?

[asynkayo]

@Sija yes sorry. Didn't find that one.
This is still happening with 0.23.1 though...

[Sija]

@Booss checkout master, it should work.

[asynkayo]

Tried on master and got this:

Using compiled compiler at `.build/crystal'
Invalid memory access (signal 11) at address 0x1c
[0x564ad8e73576] *CallStack::print_backtrace:Int32 +118
[0x564ad8e52f66] __crystal_sigfault_handler +70
[0x564ad9b07e59] sigfault_handler +40
[0x7f309df8f7e0] ???
[0x7f309fb98f80] ???
[0x7f309fbb4b0d] _ZN4llvm10DwarfDebug25finishVariableDefinitionsEv +45
[0x7f309fbc3626] _ZN4llvm10DwarfDebug18finalizeModuleInfoEv +70
[0x7f309fbc3a5a] _ZN4llvm10DwarfDebug9endModuleEv +26
[0x7f309fb8db8f] _ZN4llvm10AsmPrinter14doFinalizationERNS_6ModuleE +591
[0x7f309f5f6db9] _ZN4llvm13FPPassManager14doFinalizationERNS_6ModuleE +73
[0x7f309f603145] _ZN4llvm6legacy15PassManagerImpl3runERNS_6ModuleE +885
[0x7f30a07a573e] ???
[0x7f30a07a5be4] LLVMTargetMachineEmitToFile +244
[0x564ad98f01c9] *LLVM::TargetMachine#emit_to_file<LLVM::Module, String, LLVM::CodeGenFileType>:Bool +89
[0x564ad98f015e] *LLVM::TargetMachine#emit_obj_to_file<LLVM::Module, String>:Bool +46
[0x564ad9a257e2] *Crystal::Compiler::CompilationUnit#compile_to_object:(LLVM::Module | Nil) +834
[0x564ad9a25323] *Crystal::Compiler::CompilationUnit#compile:(LLVM::Module | Nil) +51
[0x564ad990deee] *Crystal::Compiler#codegen<Crystal::Program, Array(Crystal::Compiler::CompilationUnit), String, String>:Tuple(Array(Crystal::Compiler::CompilationUnit), Array(String)) +622
[0x564ad990d1d6] *Crystal::Compiler#codegen<Crystal::Program, Crystal::ASTNode+, Array(Crystal::Compiler::Source), String>:(Tuple(Array(Crystal::Compiler::CompilationUnit), Array(String)) | Nil) +1302
[0x564ad990f7d3] *Crystal::Compiler#compile<Array(Crystal::Compiler::Source), String>:Crystal::Compiler::Result +179
[0x564ad9a7a4f9] *Crystal::Command::CompilerConfig#compile:Crystal::Compiler::Result +57
[0x564ad9a7a4a4] *Crystal::Command::CompilerConfig#compile:Crystal::Compiler::Result +36
[0x564ad8f47482] *Crystal::Command#build:Crystal::Compiler::Result +290
[0x564ad8f4676b] *Crystal::Command#run:(Bool | Crystal::Compiler::Result | IO::FileDescriptor | Nil) +395
[0x564ad8f4650b] *Crystal::Command::run<Array(String)>:(Bool | Crystal::Compiler::Result | IO::FileDescriptor | Nil) +27
[0x564ad8f464cb] *Crystal::Command::run:(Bool | Crystal::Compiler::Result | IO::FileDescriptor | Nil) +27
[0x564ad8e3dfec] ???
[0x564ad8e66899] *_crystal_main<Int32, Pointer(Pointer(UInt8))>:Nil +9
[0x564ad97aace6] *Crystal::main_user_code<Int32, Pointer(Pointer(UInt8))>:Nil +6
[0x564ad97aab92] *Crystal::main<Int32, Pointer(Pointer(UInt8))>:Int32 +50
[0x564ad8e493f6] main +6
[0x7f309d37f4ca] __libc_start_main +234
[0x564ad8e3cf7a] _start +42
[0x0] ???

[RX14]

@Booss what LLVM version? (bin/crystal -v)

[asynkayo]

I'm using LLVM 5.0

Crystal 0.24.0+50 [c9fac56] (2017-11-13)

LLVM: 5.0.0
Default target: x86_64-unknown-linux-gnu

[rdp]

Do you still get this? Does it give you a new trace with newer crystal?