Add _replicate support to SG admin REST API

[zgramana]

As an admin, I should be able to replicate two or more Sync Gateway instances with each other.

Additional criteria:

• User and Role docs support. Not in initial delivery.
• Channel filtering.
• Cookie auth support.
• Include management API _active_tasks.

Open questions:

• Web Sockets? HTTP/2?
• Prioritization of specific replications?
• New webhooks: create, delete, started, stopped, retrying?
• Local docs?
• TTLs?
• Optimized backfills?

[adamcfraser]

I don't think we can support user and role doc replication between Sync Gateways without an enhancement to manage those docs using revision trees. It would be relatively straightforward to include user and role docs in an admin _changes request, but without rev trees for those docs, we still wouldn't have a way to resolve when updates to the user docs are made in both systems, or even avoid an echo/feedback loop when an update is originally made in only one system.

[atom992]

@zgramana Can you explain why SG need to replicate two or more Sync Gateway instances with each other.

[tleyden]

There are lots of use cases .. A simple one would be a high availability scenario with an up to date backup in a different data center.

Master-master replication also an option .. One SG on west coast and one on east coast, and user devices would connect to the closest one

[adamcfraser]

Current work is on feature/issue_1663_replicate.

Status (initial implementation):

• API for _replicate and _active_tasks stubbed out
• Start one-shot synchronous replication
• Start continuous replication
• Cancel an active replication
• active_tasks
• Properly populate response for _replicate request, excluding history
• Add counters inside replication to provide stats in history/active_tasks (moved to Add replicate metrics to replication object for use in active_tasks response #1672)
• Identify duplicate replication requests, and return 208 for existing
• Async one-shot replication, returning 202
• Channel filtering (moved to Add channel filtering for _replicate #1671)