Rspec don't exit by itself if specs passed

[etehtsea]

I've found very strange issue:

require 'rspec'
Dir.glob('lib/*.jar').each { |j| $CLASSPATH << j }

class Command < com.netflix.hystrix.HystrixCommand
  module Setter
    java_import com.netflix.hystrix.HystrixCommandGroupKey
    java_import com.netflix.hystrix.HystrixCommandProperties
    java_import com.netflix.hystrix.HystrixCommand
    java_import com.netflix.hystrix.HystrixCommandKey

    def self.build(name, group, timeout)
      HystrixCommand::Setter.withGroupKey(HystrixCommandGroupKey::Factory.asKey(group))
        .andCommandKey(HystrixCommandKey::Factory.asKey(name))
        .andCommandPropertiesDefaults(HystrixCommandProperties::Setter().withExecutionIsolationThreadTimeoutInMilliseconds(timeout))
    end
  end

  def initialize
    super(Setter.build('name', 'default', 500))
  end

  def run
    true
  end
end


describe 'NoExit' do
  specify do
    Command.new.toObservable.toBlocking.toFuture
    expect(true).to eq(true)
  end
end

describe 'Exit' do
  specify do
    Command.new.toObservable.toBlocking.toFuture
    expect(false).to be(true)
  end
end

If some of specs fails rspec successfully exit:

/tmp/rspec-exit [ master: ✔ ] 1m ⚡ bundle exec rspec -e "Exit" rspec_exit_spec.rb                                                                            jruby-1.7.18 ☺
Run options: include {:full_description=>/Exit/}
log4j:WARN No appenders could be found for logger (com.netflix.config.sources.URLConfigurationSource).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
.F

Failures:

  1) Exit should equal true
     Failure/Error: expect(false).to be(true)

       expected true
            got false
     # ./rspec_exit_spec.rb:38:in `(root)'

Finished in 0.121 seconds (files took 0.229 seconds to load)
2 examples, 1 failure

Failed examples:

rspec ./rspec_exit_spec.rb:36 # Exit should equal true

If all specs passed it won't exit until I interrupt it manually by Ctrl-C Ctrl-C.

/tmp/rspec-exit [ master: ✔ ] 8m ⚡ bundle exec rspec -e "NoExit" rspec_exit_spec.rb                                                                          jruby-1.7.18 ☺
Run options: include {:full_description=>/NoExit/}
log4j:WARN No appenders could be found for logger (com.netflix.config.sources.URLConfigurationSource).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
.

Finished in 0.119 seconds (files took 0.251 seconds to load)
1 example, 0 failures

^C
RSpec is shutting down and will print the summary report... Interrupt again to force quit.
^C%

I've fixed it in non-obvious manner:

From cd19ab8ebfe74a5868fff7e569c39e09f651d08e Mon Sep 17 00:00:00 2001
From: Konstantin Shabanov <etehtsea@gmail.com>
Date: Thu, 2 Apr 2015 02:53:12 +0600
Subject: [PATCH] Call exit even for success op

---
 lib/rspec/core/runner.rb | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lib/rspec/core/runner.rb b/lib/rspec/core/runner.rb
index af5612b..9988939 100644
--- a/lib/rspec/core/runner.rb
+++ b/lib/rspec/core/runner.rb
@@ -36,7 +36,7 @@ def self.autorun
       def self.invoke
         disable_autorun!
         status = run(ARGV, $stderr, $stdout).to_i
-        exit(status) if status != 0
+        exit(status)
       end

       # Run a suite of RSpec examples. Does not exit.

but I'm not sure that this is rspec problem at all.

/tmp/rspec-exit [ master: ✔ ] 6m ⚡ bundle exec rspec -e "NoExit" rspec_exit_spec.rb                                                                          jruby-1.7.18 ☺
Run options: include {:full_description=>/NoExit/}
log4j:WARN No appenders could be found for logger (com.netflix.config.sources.URLConfigurationSource).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
.

Finished in 0.109 seconds (files took 0.228 seconds to load)
1 example, 0 failures

Demo: https://github.com/etehtsea/rspec-exit-issue

[myronmarston]

Thanks for putting together such a great repro! That's definitely helpful.

The fact that RSpec does not call exit when all specs pass is by design. I've put together a gist showing a bug that results if we do call exit(0) when all specs pass:

https://gist.github.com/myronmarston/9ff026166f30a55ba46c

Basically, the issue is that other things (such as simplecov) may also call exit(non_zero) in order to signal that a check they do failed, and if RSpec was to call exit(0), it could stomp that and wrongly report that the process ran successfully. If you clone the gist and run the commands shown in the README, you'll see the bug.

For your specific repro case, I managed to get it pared down to something even more minimal:

# rspec_exit_spec.rb
Dir.glob('lib/*.jar').each { |j| $CLASSPATH << j }

class Command < com.netflix.hystrix.HystrixCommand
  def initialize
    key = com.netflix.hystrix.HystrixCommandGroupKey::Factory.asKey('default')
    super(key)
  end
end

describe 'NoExit' do
  specify do
    Command.new.toObservable.toBlocking.toFuture
    expect(true).to eq(true)
  end
end

describe 'Exit' do
  specify do
    Command.new.toObservable.toBlocking.toFuture
    expect(false).to be(true)
  end
end

The issue still happens with this, so this helps show that it's isolated to com.netflix.hystrix.HystrixCommand. In addition, if you comment out the toFuture call on the end of the Command.new.toObservable.toBlocking.toFuture method chain, you'll see that the issue goes away -- so that shows that calling toFuture does something to prevent the process from exiting. I believe it must be a bug in the hystrix library or in JRuby itself. I'm not a java programmer and I don't have the time to look into this further but I would guess there are good java tools to be able to sample the java stack trace at certain intervals -- maybe you can use that to see what code is running when the process gets stuck after RSpec is finished?

Closing since I don't believe this is an RSpec bug. Good luck!

[etehtsea]

@myronmarston thank you for the explanation

[etehtsea]

just FYI, this Netflix/Hystrix#760 was the issue.