Going to Production

RabbitMQ & Redis

Your RabbitMQ instance must be shared among your publishers and subscribers. Make sure you use a highly available setup, and a TCP load balancer in front of it if necessary.

Although Promiscuous allows you to use a shared Redis instance for all your applications, we recommend that each application have its own Redis instance. Given an application, in the case of a unique subscriber worker, having the Redis instance and the worker on the same machine improves performance.

Initial Sync

When deploying a new subscriber, it is useful to synchronize its database with the publisher database. Promiscuous provides two ways of synchronizing the subscriber database.

Synchronizing is also useful when adding new published attributes. Although we consider it a bad practice, note that Promiscuous doesn't support changes in class types yet (e.g. a Admin instance changes to be a Member instance).

Programmatically

User.each { |u| u.promiscuous.sync }

From the command line

bundle exec promiscuous publish UserGroup "User.where(:updated_at.gt => 1.day.ago)"

Promiscuous will publish each of these collections with a progress bar in the given order. Note that you can partition your data with the right selectors to publish with many workers. Partition overlapping is fine.

Error Handling

Promiscuous tries to reconnect every 2 seconds when the connection to RabbitMQ or Redis has been lost. During this time, the publisher cannot perform any write queries. To take a publisher instance out of the load balancer, Promiscuous provides a health checker. Promiscuous.healthy? returns true when the system is able to publish or subscribe to messages.

On the subscriber side, when a message cannot be processed due to a raised exception, for example because of a database failure, the message will be retried with an exponential backoff.

When errors occur, the configurable error_notifier is invoked with the following exceptions:

Promiscuous::Error::Connection # Lost connection
Promiscuous::Error::Publisher  # Failed to publish a model operation
Promiscuous::Error::Subscriber # Failed to processing a message
Promiscuous::Error::Recover    # Deadlock recovery, Skipped messages

More details of their internals are in the code, but most messages will make sense. Exceptions are logged as well.

Instrumentation

Promiscuous supports NewRelic. Just add the promiscuous-newrelic gem to your Gemfile to instrument the performance of the subscriber workers:

gem 'promiscuous-newrelic'

For NewRelic error support, hook the error_notifier:

Promiscuous.configure do |config|
  config.error_notifier = proc do |exception|
    NewRelic::Agent.notice_error(exception)
  end
end

For Airbrake support, hook the error_notifier:

Promiscuous.configure do |config|
  config.error_notifier = proc do |exception|
    Airbrake.notify(exception)
  end
end

Monitor closely the size of the RabbitMQ queues (ready messages).

Managing Deploys

When adding new attributes on the publisher and subscriber sides, you must deploy the publisher first, then the subscriber. Doing otherwise may block the subscriber as it wouldn't be able to process a message with missing attributes. In this case, you would have to rollback the subscriber deployment, restart the worker, wait until fresh messages are getting in, and re-deploy the subscriber.

Setup with Unicorn, Resque, etc.

When unicorn or resqeue forks (booh!), you need to disconnect/reconnect Promiscuous to ensure each child has its own connection.

Example with Unicorn:

before_fork do
  Promiscuous.disconnect
end

after_fork do
  Promiscuous.connect
end

Example with Resque:

lib/tasks/resque.rb:

require "resque/tasks"
task "resque:setup" => :environment do
  Resque.before_first_fork = proc do
    Promiscuous.disconnect
  end

  Resque.after_fork = proc do
    Promiscuous.connect
  end
end