Home

Getting stuff running

Rails file cache -- by eLobato

Concurrent access to cache files by different process can easily make your tests fail. Make each process have its own cache file store to avoid this problem.

Add to config/environments/test.rb:

config.cache_store = :file_store, Rails.root.join("tmp", "cache", "paralleltests#{ENV['TEST_ENV_NUMBER']}")

Rails Sprockets file cache -- by jordan-brough

Sprockets uses a default cache path of <rails-root>/tmp/cache/assets/<environment>. Concurrent usage of the same cache directory can cause sporadic problems like "ActionView::Template::Error: end of file reached".

Add to config/environments/test.rb:

if ENV['TEST_ENV_NUMBER']
  assets_cache_path = Rails.root.join("tmp/cache/assets/paralleltests#{ENV['TEST_ENV_NUMBER']}")
  Rails.application.config.assets.cache = Sprockets::Cache::FileStore.new(assets_cache_path)
end

With sphinx -- by ujh

At file config/sphinx.yml in the test section:

test:
  mysql41: <%= 9313 + ENV['TEST_ENV_NUMBER'].to_i %>
  indices_location: <%= File.join(Rails.root, "db", "sphinx", "test#{ENV['TEST_ENV_NUMBER']}") %>
  configuration_file: <%= File.join(Rails.root, "config", "test#{ENV['TEST_ENV_NUMBER']}.sphinx.conf")%>
  log: <%= File.join(Rails.root, "log", "test#{ENV['TEST_ENV_NUMBER']}.searchd.log") %>
  query_log: <%= File.join(Rails.root, "log", "test#{ENV['TEST_ENV_NUMBER']}.searchd.query.log") %>
  binlog_path: <%= File.join(Rails.root, "tmp", "binlog", "test#{ENV['TEST_ENV_NUMBER']}") %>
  pid_file: <%= File.join(Rails.root, "tmp", "pids", "test#{ENV['TEST_ENV_NUMBER']}.sphinx.pid") %>

If solution above doesn't work, then you may try another approach via pat's recommendation:

test:
  mysql41: <%= ENV['TEST_ENV_NUMBER'].to_i + 9307 %>
  pid_file: <%= File.join(Rails.root, "tmp", "searchd.#{ENV['TEST_ENV_NUMBER']}.pid") %>
  indices_location: <%= File.join(Rails.root, "db", "sphinx", "#{ENV['TEST_ENV_NUMBER']}") %>
  configuration_file: <%= File.join(Rails.root, "config", "test.#{ENV['TEST_ENV_NUMBER']}.sphinx.conf") %>
  binlog_path: <%= File.join(Rails.root, "db", "sphinx", "#{ENV['TEST_ENV_NUMBER']}", "binlog") %>

With capybara(~>0.4.0)+selenium -- by rgo

Capybara.server_port = 9887 + ENV['TEST_ENV_NUMBER'].to_i

With capybara(=0.3.9)/Rails 2.3 -- by xunker

Add to features/support/env.rb:

if ENV['TEST_ENV_NUMBER']
  class Capybara::Server
    def find_available_port
      @port = 9887 + ENV['TEST_ENV_NUMBER'].to_i
      @port += 1 while is_port_open?(@port) and not is_running_on_port?(@port)
    end
  end
end

With Selenium in Docker -- by ZimbiX

In order to avoid browser crashes due to /dev/shm (shared memory) being too small, specify a larger shm_size.

If you're running Docker-in-Docker, using the alternative of mounting /dev/shm won't work unless you increase the shm_size of the outer container.

e.g., with Docker Compose:

services:
  selenium:
    image: selenium/standalone-chrome:3.141.59@sha256:d0ed6e04a4b87850beb023e3693c453b825b938af48733c1c56fc671cd41fe51
    shm_size: 1G

With rspec_junit_formatter -- by jgarber

I've had better results with rspec_junit_formatter than with ci_reporter. Parallelizing it is easy!

Add this to .rspec_parallel:

--format RspecJunitFormatter
--out tmp/rspec<%= ENV['TEST_ENV_NUMBER'] %>.xml

Then you can configure your CI to publish the JUnit test result report files.

• Jenkins: use tmp/rspec*.xml for "Test report XMLs".
• Semaphore 2.0: can also handle multiple files, eg test-results publish tmp/rspec*.xml (see docs for more detail on setting up).
• GitLab-CI: See the gradle example

@danielheath : I've had corrupted xml files with larger numbers of concurrent processes - not sure this is reliable.

dacook: to make all specs appear in the same suite, here's a quick transform to give them all the same name:

sed -i 's/name="rspec[0-9]"/name="rspec"/' tmp/junit*.xml # rename so they get combined

With ci_reporter for rspec -- by morganchristiansson

export CI_REPORTS=results

Add spec/parallel_specs.opts with the contents:

--format progress
--require ci/reporter/rake/rspec_loader
--format CI::Reporter::RSpec:/dev/null

Our project has the following in test/test_helper.rb

if ENV["CI_REPORTS"] == "results"
  require "ci/reporter/rake/test_unit_loader"
end

Run the tasks like this:

rake "parallel:features[,,--format progress --format junit --out ${CI_REPORTS} --no-profile -r features]"

Or without rake like this:

bundle exec $(bundle show parallel_tests)/bin/parallel_test --type features -o '--format progress --format junit --out ${CI_REPORTS} --no-profile -r features'

with .rspec_parallel

Add following to RAILS_ROOT/.rspec_parallel.

--format progress
--require ci/reporter/rake/rspec_loader
--format CI::Reporter::RSpec

Run rake parallel:spec, then rspec generates reports in spec/reports.

For more information on how to configure ci_reporter check under advanced usage on http://caldersphere.rubyforge.org/ci_reporter/

With ci_reporter for test_unit -- by phoet

See this issue 29 for more information:

# add the ci_reporter to create reports for test-runs, since parallel_tests is not invoked through rake
puts "running on #{Socket.gethostname}"
if /buildserver/ =~ Socket.gethostname
  require 'ci/reporter/test_unit'
  module Test
    module Unit
      module UI
        module Console
          class TestRunner
            def create_mediator(suite)
              # swap in ci_reporter custom mediator
              return CI::Reporter::TestUnit.new(suite)
            end
          end
        end
      end
    end
  end
end

With DatabaseCleaner for RSpec -- by sciprog

See issue 66 for more information.

Do not use the truncation strategy in DatabaseCleaner, in your RSpec config.
This strategy seems to cause a bottleneck which will negate any gain made
through parallelization of your tests. If possible, use the transaction strategy
over the truncation strategy.
    *Note: This issue does not seem to exist in relation to features, only to specs.

Features not running in subdirectories

If you have put your features into subdirectories you may have problems running them in parallel as it will not find your step_definitions. To work around this I put all my features into the features directory.

You have to require features/ folder for cucumber in order to load step definitions.

 rake "parallel:features[4, '', '-rfeatures/']"

If you want it to work with rake parallel:features add -rfeatures/ to the end of std_opts in config/cucumber.yml

Using Cucumber release 3 --retry

rake parallel:features[,, --retry 1]

• 1 digit used for the number of times you want to retry Rerun file no longer required

Disable parallel run for certain cases (cucumber)

If you do not have enough scenarios to warrant a separate instance of external service as described in sphinx section above, create a "mutex" in your env.rb file

Before("@solr") do
  #we do not want solr tests to run in parallel, so let's simulate a mutex
  while File.exists?("tmp/cucumber_solr")
    sleep(0.2)
  end
  File.open("tmp/cucumber_solr", "w") {}
  Sunspot.session = $original_sunspot_session
  Sunspot.remove_all!
  # or do other things
end

After("@solr") do
  File.delete("tmp/cucumber_solr")
end

Fixing "unable to bind to locking port 7054 within 45 seconds ..." error

When running with cucumber + capybara + selenium-webdriver (w firefox), this error may be encountered due to firing of too many firefox instances altogether. To fix this issue, add the following to features/support/env.rb:

unless (env_no = ENV['TEST_ENV_NUMBER'].to_i).zero?
  # As described in the readme
  Capybara.server_port = 8888 + env_no

  # Enforces a sleep time, i need to multiply by 10 to achieve consistent results on
  # my 8 cores vm, may work for less though.
  sleep env_no * 10
end

With action_mailer_cache_delivery (~> 0.3.2) -- by p0deje

You may get unexpected errors like EOFError. If so, make sure cache files differ for processes. Change you config/environment/test.rb

config.action_mailer.cache_settings = { :location => "#{Rails.root}/tmp/cache/action_mailer_cache_delivery#{ENV['TEST_ENV_NUMBER']}.cache" }

With RSpec + Sunspot

It's best to use with sunspot-rails-tester. Then you don't need to run solr with rake task. For this you will only have to update config/sunspot.yml you can make it like this:

test:
  solr:
    hostname: localhost
    port: <%= 8981 + ENV['TEST_ENV_NUMBER'].to_i %>
    log_level: WARNING
    data_path: <%= File.join(::Rails.root, 'solr', 'data', ::Rails.env, ENV['TEST_ENV_NUMBER'].to_i.to_s) %>

With RSpec + Sunspot(>=2.2.0)

It's best to use with sunspot-rails-tester. Then you don't need to run solr with rake task. For this you will only have to update config/sunspot.yml you can make it like this:

test:
  solr:
    port: <%= 8981 + ENV['TEST_ENV_NUMBER'].to_i %>
    solr_home: <%= File.join(::Rails.root, 'solr', 'data', ::Rails.env, ENV['TEST_ENV_NUMBER'].to_i.to_s) %>

With TeamCity -- by aaronjensen

TeamCity has its own logger so you'll need to use the --serialize-stdout flag when you run anything in parallel. You'll need to use a custom rake task or script and call parallel_rspec/parallel_test/parallel_cucumber directly.

You will need this rake task so that TeamCity will calculate the number of specs correctly. However what it does not pick up is duplicated test names within a spec. TeamCity processes this as a duplicate and does not add it to the total spec count.

namespace :teamcity do
  task parallel_rspec: :environment do
    sh('parallel_rspec spec --serialize-stdout')
  end
end

With SPORK

Spork helps minimize the rails load time per-core, only performing it once. On an 8-virtual core rMBP this saved 20 seconds, running 1000 specs that used to take 65 seconds in just under 15 seconds.

The problem is spork seems not to use the separate database instances configured by rake parallel:create so we get a slew of DB lock issues. However, if we configure rspec to use in-memory databases, this problem goes away. This post shows how, but basically just add

setup_sqlite_db = lambda do
  ActiveRecord::Base.establish_connection(adapter: 'sqlite3', database: ':memory:')
  load "#{Rails.root.to_s}/db/schema.rb" # use db agnostic schema by default
end
silence_stream(STDOUT, &setup_sqlite_db)

to the bottom of your spec_helper.rb, and voila!

Gotcha (Conflict with Simplecov and json)

parallel_tests, simplecov, and json do not seem to play well together. Caused a number of errors in the form:

/.rvm/gems/ruby-1.9.3-p484@mercury/gems/json-1.8.1/lib/json/common.rb:155:in `parse': 795: unexpected token at 'null, (MultiJson::LoadError)

Until a better workaround is found, remove simplecov from your project.

Custom script for parallel_test --exec which splits a set of files

Note: PARALLEL_TEST_GROUPS is an environment variable which has the number of test groups in use. It is unset if running without parallel_test.

#!/bin/bash
# custom_parallel_script.sh
# Usage: parallel_test --exec ./custom_parallel_script.sh

FILES=$(find . -type f -name \*.rb)

# workaround for the fact that TEST_ENV_NUMBER is '' for the 1st group - default to 1 if unset
LOCAL_TEST_ENV_NUMBER=${TEST_ENV_NUMBER:-1}

i=0
for f in $FILES; do
  if [[ -z "$PARALLEL_TEST_GROUPS" || $(($i % $PARALLEL_TEST_GROUPS)) -eq $LOCAL_TEST_ENV_NUMBER ]]; then
    echo $f  # real action here
  fi
  ((i++))
done

Tear down after all processes are done (RSpec) -- by demental

Say your specs rely on building a js app before suite, and build files must be deleted after suite is finished. before(:suite) will run after the first process is done, so all the other tests running after that are likely to fail. As a workaround, you can create a task and redefine rake parallel:spec

require_relative '../../spec/support/test_tools.rb'

namespace :parallel do
  desc "Run parallel spec Clear artifact after parallel testing"
  task :clean_spec => [:spec, :clean]

  desc "Remove artifacts after parallel tests"
  task :clean do
    TestTools.remove_build_files
  end
end

# In spec_helper.rb
config.after(:suite) do
  TestTools.remove_build_files unless ENV.key? 'TEST_ENV_NUMBER'
end

# spec/support/test_tools.rb
module TestTools
  def self.remove_build_files
    # Doing the cleanup here...
    FileUtils.rm_rf(Rails.root.join('tmp','test_build'))
  end
end

If running regular rspec command, cleanup will be done after(:suite) as usual. Otherwise, it will be run after rake task parallel:spec is finished. Credits goes to seuros for this workaround.

With searchkick -- by emaxi

Set index name based in TEST_ENV_NUMBER for each model using searchkick, example app/model/product.rb:

class Product < ActiveRecord::Base
  searchkick index_name: "products#{ENV['TEST_ENV_NUMBER']}"
end

You can also override the .env via an initializer:

if Rails.env.test?
  Searchkick.env = "test#{ENV['TEST_ENV_NUMBER']}"
end

In 2.2.1 or more you can use Searchkick.index_suffix

Searchkick.index_suffix = ENV["TEST_ENV_NUMBER"]

https://github.com/ankane/searchkick#parallel-tests

With poltergeist -- by emaxi

Use different port for each process spec/spec_helper.rb:

Capybara.register_driver :poltergeist do |app|
  options = {
    port: 51674 + ENV['TEST_ENV_NUMBER'].to_i
  }
  Capybara::Poltergeist::Driver.new(app, options)
end

With mongoid -- by emaxi

At file config/mongoid.yml in the test section:

Mongoid >= 5.x

test:
  sessions:
    default:
      database: app_name_test<%= ENV['TEST_ENV_NUMBER'] %>
      hosts:
        - localhost:27017

Mongoid <= 4.x

test:
  client:
    default:
      database: app_name_test<%= ENV['TEST_ENV_NUMBER'] %>
      hosts:
        - localhost:27017

With headless -- by sauliusgrigaitis

Headless.new(display: 100, reuse: true, destroy_at_exit: false).start

With simplecov -- by a grateful user

To print the simplecov report after ALL threads are finished, to stop printing for each thread, do:

spec_helper.rb

if ENV['COVERAGE'] == 'true'
  require 'simplecov'
  require 'simplecov-console'
  SimpleCov.formatter = SimpleCov::Formatter::Console
  SimpleCov.start 'rails'
  if ENV['TEST_ENV_NUMBER'] # parallel specs
    SimpleCov.at_exit do
      result = SimpleCov.result
      result.format! if ParallelTests.number_of_running_processes <= 1
    end
  end
end

RSpec.configure do |config|
 ...

gems:

group :test do
  gem 'simplecov'
  gem 'simplecov-console'

Extras: Reduce noise in the output:

spec_helper.rb or rails_helper.rb

RSpec.configure do |config|
  ...
  # mute noise for parallel tests
  config.silence_filter_announcements = true if ENV['TEST_ENV_NUMBER']

With elasticsearch-extensions -- by [thatandromeda]

If you expect to be starting test clusters in parallel, the options with which you initialize them (Elasticsearch::Extensions::Test::Cluster.start(**es_options)) must include the following:

es_options = {
    port: 9250 + ENV['TEST_ENV_NUMBER'].to_i,
    cluster_name: "cluster#{ENV['TEST_ENV_NUMBER']}",
    path_data: "/tmp/elasticsearch_test#{ENV['TEST_ENV_NUMBER']}"
  }

Isolating the ports isn't enough as the nodes you create with each start will find each other, causing the health checks on cluster startup to fail as the number of nodes in the cluster won't match the number of nodes specified in its arguments. You have to separate the clusters so they can't find one another.

with redis -- by yoav

Redis supports up to 16 databases (0 to 15). If you are clearing the cache before or after every test, make sure that each process is using its own redis database.

db = ENV['TEST_ENV_NUMBER'].nil? ? 1 : (ENV['TEST_ENV_NUMBER'].presence || '1').to_i - 1
Redis.new(db: db)

Also, make sure to use redis.flushdb and not redis.flushall

with carrierwave -- by yoav

removing all files, in the uploads directory, before or after each test will break other processes. Instead, remove all files before the suite is run.

with activestorage

In config/storage.yml, modify root to include the test env number:

test:
  service: Disk
  root: <%= Rails.root.join("tmp/storage#{ENV['TEST_ENV_NUMBER']}") %>

---

!! Add your own experience / gotchas !!