Deployment

Overview

To safely shut down Sidekiq, you need to send it the TSTP signal as early as possible in your deploy process and the TERM signal as late as possible. TSTP tells Sidekiq to stop pulling new work and finish all current work. TERM tells Sidekiq to exit within N seconds, where N is set by the -t timeout option and defaults to 25. Using TSTP+TERM in your deploy process gives your jobs the maximum amount of time to finish before exiting.

If any jobs are still running when the timeout is up, Sidekiq will push those jobs back to Redis so they can be rerun later.

Your deploy scripts must give Sidekiq N+5 seconds to shutdown cleanly after the TERM signal. For example, if you send TERM and then send KILL 10 seconds later, you can lose jobs (if using Sidekiq) or see duplicate job execution (if using Sidekiq Pro's super_fetch).

Network Architecture

I recommend running 1 or more Sidekiqs per app server in your production cluster. At a previous employer, I ran two Sidekiq processes on three app servers, for six total processes. With a concurrency of 10, this gives you 60 worker threads. All six processes talked to the same Redis server and we used high, default and low queues to keep queue management and job priority as simple as possible.

Running separate worker machines for Sidekiq or using many different queues adds complexity where it was not needed for us.

Heroku

Heroku has its own page now.

Running your own Process

If you want to run Sidekiq on your own server, use systemd to start Sidekiq as a system service. This will ensure the process is restarted if Sidekiq crashes.

Here's an example systemd unit file. The systemd documentation has sections on the .service file and executing processes. Every developer who uses Linux should read these pages many times.

systemctl stop sidekiq
systemctl start sidekiq
systemctl restart sidekiq
systemctl kill -s TSTP sidekiq # quiet

Sidekiq Enterprise's sidekiqswarm binary makes it trivial to start N Sidekiq processes with a single systemd service. Take a look at Foreman for a useful tool to automate your application processes in development and production.

Capistrano

Use the capistrano-sidekiq gem (github). Integrated support has been removed. Warning: Capistrano uses daemonization by default so if the Sidekiq process crashes, it will not restart automatically.

Events

Sidekiq fires process lifecycle events when starting up and shutting down:

Sidekiq.configure_server do |config|
  # runs after your app has finished initializing but before any jobs are dispatched.
  config.on(:startup) do
    make_some_singleton
  end
  config.on(:quiet) do
    puts "Got TSTP, stopping further job processing..."
  end
  config.on(:shutdown) do
    puts "Got TERM, shutting down process..."
    stop_the_world
  end
end

This can be extremely useful if you want to start/stop your own threads/actors or signal.

Sidekiq Pro users can use :quiet to signal long-running jobs that they need to prepare for an upcoming restart (this requires super_fetch to be enabled).

# config/initializers/sidekiq.rb
$shutdown_pending = false
Sidekiq.configure_server do |config|
  config.on(:quiet) do
    $shutdown_pending = true
  end
end

# some_job.rb
def perform
  # This job might process 1000s of items and take an hour.
  # Have each iteration check for shutdown. big_list_of_items
  # should only return unprocessed items so the loop will continue
  # where it left off.
  big_list_of_items.find_each do |item|
    process(item)
    # Sidekiq Pro will restart job immediately on process restart
    raise Sidekiq::Shutdown if $shutdown_pending
  end
end

Previous: Delayed Extensions Next: Monitoring