Ent Leader Election

Sidekiq Enterprise will automatically elect a leader from the set of running processes, aka the "Senate". The leader is extremely useful when you want to coordinate work across the entire cluster without duplication of effort.

The Election Process

The first time a Sidekiq Enterprise process starts, it will check Redis for the current leader. If the leader's info is current, the process will automatically configure itself as a follower. If there's no leader information, the process will attempt a coup and elect itself leader. Viva la revolución!

There's no need for Raft, Paxos or other fancy algorithms because there's no distributed consensus going on here: Redis determines who is leader.

Leaders renew and verify their leadership every 15 seconds. Followers verify there is a leader every 60 seconds. Followers are prepared to take over as leader during that verification check. If the leader loses connectivity to Redis, its leadership will expire and a follower with connectivity will take over as leader.

Leader processes automatically "step down" when they exit, ensuring that another leader is elected promptly when deploying.

Running Code on the Leader

You can start your own threads within Sidekiq and use the Leader API to determine if your thread is running within the Leader process. Here's am example of creating your own custom service within Sidekiq 7.x.

# Define a service which does something within the Leader process
# every 30 seconds.
class MyService
  include Sidekiq::Component

  def initialize(config)
    @config = config
    @done = false
  end

  def start
    @thread = safe_thread("myservice", &method(:process))
  end

  def stop
    @done = true
  end

  private

  def process
    until @done
      if leader?
        # I am the leader, do something
        # redis {|c| c.get("some-key") }
        logger.info "ping"
      end
      sleep 30
    end
  end
end

Sidekiq.configure_server do |config|
  config.on(:startup) do
    config.lookup("myservice", MyService).start
  end
  config.on(:quiet) do
    config.lookup("myservice")&.stop
  end
end