Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Event Machine SegFaults #237

Closed
AlanFoster opened this issue Apr 16, 2018 · 6 comments
Closed

Event Machine SegFaults #237

AlanFoster opened this issue Apr 16, 2018 · 6 comments

Comments

@AlanFoster
Copy link
Contributor

AlanFoster commented Apr 16, 2018 •
edited
Loading

Hey; I've been running into some segfaults with Event Machine:

/usr/local/bundle/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:202: [BUG] Segmentation fault at 0x0000000000000000
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0006 p:---- s:0026 e:000025 CFUNC :release_machine
c:0005 p:0036 s:0022 e:000021 RESCUE /usr/local/bundle/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:202
c:0004 p:0385 s:0018 e:000017 METHOD /usr/local/bundle/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:205
c:0003 p:0015 s:0010 E:000330 METHOD /usr/local/bundle/bundler/gems/puffing-billy-ee2654d63abe/lib/billy/proxy.rb:44
c:0002 p:0008 s:0006 e:000005 BLOCK /usr/local/bundle/bundler/gems/puffing-billy-ee2654d63abe/lib/billy/proxy.rb:18 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/usr/local/bundle/bundler/gems/puffing-billy-ee2654d63abe/lib/billy/proxy.rb:18:in `block in start'
/usr/local/bundle/bundler/gems/puffing-billy-ee2654d63abe/lib/billy/proxy.rb:44:in `main_loop'
/usr/local/bundle/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:205:in `run'
/usr/local/bundle/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:202:in `ensure in run'
/usr/local/bundle/gems/eventmachine-1.2.0.1/lib/eventmachine.rb:202:in `release_machine'

I believe this was caused by an event machine version bump introduced here:
bc6aae5

However, I noticed that in the next commit the version of Event Machine was downgraded a few commits later - potentially by mistake:

b74dc54#diff-752c3d3303256b2b755956cf77df54edR33

  • gem.add_runtime_dependency 'eventmachine', '~> 1.0.4'

So how am I seeing event machine crashes? Well, I have an inflight pull request that I've been using for the past month or so #225

It turns out that I branched off the codebase that had eventmachine 1.2, which is why I'm seeing it. I believe if I rebase my against master and revert to 1.0.4 this issue will be resolved for me.

I have raised this puffing billy issue issue to:

  • Highlight that this version of Event Machine segfaults frequently with Puffing BIlly - I believe this is the specific issue Segmentation fault eventmachine/eventmachine#670
  • Highlight that the event machine version may have been reverted accidentally, but maybe this may be a blessing in disguise
@AlanFoster
Copy link
Contributor Author

Hm, rebasing my request log PR against the latest master can cause segfaults still it seems:

/usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:202: [BUG] Segmentation fault at 0x0000000000000000
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0006 p:---- s:0026 e:000025 CFUNC  :release_machine
c:0005 p:0038 s:0022 e:000021 RESCUE /usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:202
c:0004 p:0432 s:0018 e:000017 METHOD /usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:226
c:0003 p:0015 s:0010 E:000ff0 METHOD /usr/local/bundle/bundler/gems/puffing-billy-0bc70a6d12c5/lib/billy/proxy.rb:44
c:0002 p:0008 s:0006 e:000005 BLOCK  /usr/local/bundle/bundler/gems/puffing-billy-0bc70a6d12c5/lib/billy/proxy.rb:18 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/usr/local/bundle/bundler/gems/puffing-billy-0bc70a6d12c5/lib/billy/proxy.rb:18:in `block in start'
/usr/local/bundle/bundler/gems/puffing-billy-0bc70a6d12c5/lib/billy/proxy.rb:44:in `main_loop'
/usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:226:in `run'
/usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:202:in `ensure in run'
/usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:202:in `release_machine'

-- Machine register context ------------------------------------------------
 RIP: 0x0000000000000000 RBP: 0x0000000000000001 RSP: 0x00007efe160f22c8
 RAX: 0x00007efe154d1358 RBX: 0x00007efe04000cc0 RCX: 0x00007efe0405c100
 RDX: 0x00007efe0405c0f0 RDI: 0x00007efe04182450 RSI: 0x0000000000000000
  R8: 0x00007efe0400d6d0  R9: 0x0000000000000001 R10: 0x000000000236d520
 R11: 0x0000000000000000 R12: 0x0000000008e63e00 R13: 0x000000000e0fe4b0
 R14: 0x0000000008e3bf60 R15: 0x0000000008e4fbb0 EFL: 0x0000000000010206

-- C level backtrace information -------------------------------------------
/usr/local/lib/libruby.so.2.4(rb_vm_bugreport+0x4f3) [0x7efe22130563]
/usr/local/lib/libruby.so.2.4(rb_bug_context+0xd4) [0x7efe21fba824]
/usr/local/lib/libruby.so.2.4(sigsegv+0x3e) [0x7efe220a7f1e]
[0x7efe21b844b0]

This error only popped up after a Rails 5.0 upgrade, but I can't be certain if that's related. I'll keep poking further. I believe things are more stable after adding an after suite hook to manually shut down EventMachine, but I'll run the test suite continuously to confirm if that is a fix

@AlanFoster
Copy link
Contributor Author

I'm going to run with the latest version of puffing billy and see if this occurs. If it does, I'll try to investigate an explicit shutdown of event machine as part of an after suite call.

@AlanFoster
Copy link
Contributor Author

I can confirm that this can trigger with Puffing Billy 1.0.0 with Rails 5, Debian, Ruby 2.4.2

Randomized with seed 29647

/usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:202: [BUG] Segmentation fault at 0x0000000000000000
ruby 2.4.2p198 (2017-09-14 revision 59899) [x86_64-linux]

-- Control frame information -----------------------------------------------
c:0006 p:---- s:0026 e:000025 CFUNC  :release_machine
c:0005 p:0038 s:0022 e:000021 RESCUE /usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:202
c:0004 p:0432 s:0018 e:000017 METHOD /usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:226
c:0003 p:0015 s:0010 E:002330 METHOD /usr/local/bundle/gems/puffing-billy-1.0.0/lib/billy/proxy.rb:44
c:0002 p:0008 s:0006 e:000005 BLOCK  /usr/local/bundle/gems/puffing-billy-1.0.0/lib/billy/proxy.rb:18 [FINISH]
c:0001 p:---- s:0003 e:000002 (none) [FINISH]

-- Ruby level backtrace information ----------------------------------------
/usr/local/bundle/gems/puffing-billy-1.0.0/lib/billy/proxy.rb:18:in `block in start'
/usr/local/bundle/gems/puffing-billy-1.0.0/lib/billy/proxy.rb:44:in `main_loop'
/usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:226:in `run'
/usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:202:in `ensure in run'
/usr/local/bundle/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:202:in `release_machine'

-- Machine register context ------------------------------------------------
 RIP: 0x0000000000000000 RBP: 0x0000000000000004 RSP: 0x00007f4920e722c8
 RAX: 0x00007f4920251358 RBX: 0x00007f490c000cc0 RCX: 0x00007f490c2610f8
 RDX: 0x00007f490c2610d0 RDI: 0x00007f490c81f350 RSI: 0x0000000000000000
  R8: 0x00007f490c9d8620  R9: 0x0000000000000001 R10: 0x00007f4902767328
 R11: 0x0000000000000000 R12: 0x000000000789da08 R13: 0x000000000ca5f940
 R14: 0x000000000788d8e0 R15: 0x000000000789ad60 EFL: 0x0000000000010206

-- C level backtrace information -------------------------------------------
/usr/local/lib/libruby.so.2.4(rb_vm_bugreport+0x4f3) [0x7f492ceb0563]
/usr/local/lib/libruby.so.2.4(rb_bug_context+0xd4) [0x7f492cd3a824]
/usr/local/lib/libruby.so.2.4(sigsegv+0x3e) [0x7f492ce27f1e]
[0x7f492c9044b0]

@ronwsmith
Copy link
Collaborator

Eek, segfaults are the worst! How do things look with the latest EM?

@AlanFoster AlanFoster reopened this Apr 18, 2018
@AlanFoster
Copy link
Contributor Author

I believe I tried with the latest version 1.2.5, as well as other versions inbetween.

I have a fork of Puffing Billy that safely shuts down event machine via an after(:suite) hook, and I've not seen this segfault again. I'm preparing pull request currently, but I'll be running the test suite continuously for a few days to confirm this is a valid workaround first 👍

@AlanFoster AlanFoster mentioned this issue Apr 19, 2018
Merged
@sodabrew
Copy link

Hi! Please try EventMachine 1.2.6, which has an important crash fix. https://github.com/eventmachine/eventmachine/releases/tag/v1.2.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@sodabrew @AlanFoster @ronwsmith