Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restart worker fails to restart and dies on closed stream #78

Closed
dblock opened this issue Sep 18, 2018 · 5 comments
Closed

Restart worker fails to restart and dies on closed stream #78

dblock opened this issue Sep 18, 2018 · 5 comments

Comments

@dblock
Copy link
Collaborator

dblock commented Sep 18, 2018

W, [2018-09-18T04:37:55.107838 #220]  WARN -- : DOWN: name=Team, id=T..., 2 retries left
W, [2018-09-18T04:38:55.238204 #220]  WARN -- : DOWN: name=Team, id=T..., 1 retry left
W, [2018-09-18T04:39:55.361735 #220]  WARN -- : DOWN: name=Team, id=T..., 0 retries left
W, [2018-09-18T04:39:55.361934 #220]  WARN -- : RESTART: name=Team, id=T...
W, [2018-09-18T04:39:55.362279 #220]  WARN -- : Error restarting team 5996f4422a947200b1c10399: closed stream.
D, [2018-09-18T04:39:55.362360 #220] DEBUG -- : PING: name=Team, id=T..., done.
dblock added a commit to dblock/slack-ruby-bot-server that referenced this issue Sep 18, 2018
dblock added a commit to dblock/slack-ruby-bot-server that referenced this issue Sep 18, 2018
dblock added a commit to dblock/slack-ruby-bot-server that referenced this issue Sep 18, 2018
@dblock dblock closed this as completed in 746dea1 Sep 20, 2018
dblock added a commit that referenced this issue Sep 20, 2018
Fix #78: ping thread terminates on a failed restart.
@dblock
Copy link
Collaborator Author

dblock commented Sep 25, 2018

Still a problem, that didn't fix it :(

@dblock dblock reopened this Sep 25, 2018
@dblock
Copy link
Collaborator Author

dblock commented Sep 25, 2018

W, [2018-09-25T23:52:02.822982 #213]  WARN -- : Error closing driver for 568e992e012d800003000004: closed stream.
W, [2018-09-25T23:52:02.823149 #213]  WARN -- : Error restarting team 568e992e012d800003000004: closed stream.

@dblock
Copy link
Collaborator Author

dblock commented Sep 25, 2018

19:58:53 console.1 | E, [2018-09-25T19:58:53.905879 #43147] ERROR -- : closed stream (IOError)
19:58:53 console.1 | /Users/dblock/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/openssl/buffering.rb:388:in `syswrite_nonblock'
19:58:53 console.1 | /Users/dblock/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/openssl/buffering.rb:388:in `write_nonblock'
19:58:53 console.1 | /Users/dblock/.rvm/gems/ruby-2.5.1/gems/async-io-1.15.5/lib/async/io/generic.rb:128:in `async_send'
19:58:53 console.1 | /Users/dblock/.rvm/gems/ruby-2.5.1/gems/async-io-1.15.5/lib/async/io/generic.rb:47:in `block in wrap_blocking_method'
19:58:53 console.1 | /Users/dblock/.rvm/gems/ruby-2.5.1/gems/async-websocket-0.6.1/lib/async/websocket/connection.rb:93:in `write'
19:58:53 console.1 | /Users/dblock/.rvm/gems/ruby-2.5.1/gems/websocket-driver-0.7.0/lib/websocket/driver/hybi.rb:228:in `send_frame'
19:58:53 console.1 | /Users/dblock/.rvm/gems/ruby-2.5.1/gems/websocket-driver-0.7.0/lib/websocket/driver/hybi.rb:191:in `frame'
19:58:53 console.1 | /Users/dblock/.rvm/gems/ruby-2.5.1/gems/websocket-driver-0.7.0/lib/websocket/driver/hybi.rb:154:in `close'
19:58:53 console.1 | /Users/dblock/.rvm/gems/ruby-2.5.1/gems/async-websocket-0.6.1/lib/async/websocket/connection.rb:97:in `close'

@dblock
Copy link
Collaborator Author

dblock commented Sep 26, 2018

@ioquatix could use a bit of help here, I think I am not closing this thing correctly in all cases:

  1. Async::IO::SSLSocket#close
  2. Slack::RealTime::Concurrency::Async::Socket#close
  3. Slack::RealTime::Concurrency::Async::Socket#emit(close event)

This works most of the time, but sometimes ends up with IOError in 2). Looking at the order it makes sense that sending a frame over a closed SSL socket shouldn't work. But then why does it work most of the time? And what's the correct order of force-closing this connection?

@dblock
Copy link
Collaborator Author

dblock commented Feb 25, 2019

In #93 the ping worker has been removed in favor of an implementation in slack-ruby-client.

@dblock dblock closed this as completed Feb 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant