Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Remove HTTP replication APIs #2117

Merged
merged 5 commits into from
Apr 12, 2017
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 4 additions & 22 deletions docs/replication.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,28 +26,10 @@ expose the append-only log to the readers should be fairly minimal.
Architecture
------------

The Replication API
~~~~~~~~~~~~~~~~~~~

Synapse will optionally expose a long poll HTTP API for extracting updates. The
API will have a similar shape to /sync in that clients provide tokens
indicating where in the log they have reached and a timeout. The synapse server
then either responds with updates immediately if it already has updates or it
waits until the timeout for more updates. If the timeout expires and nothing
happened then the server returns an empty response.

However unlike the /sync API this replication API is returning synapse specific
data rather than trying to implement a matrix specification. The replication
results are returned as arrays of rows where the rows are mostly lifted
directly from the database. This avoids unnecessary JSON parsing on the server
and hopefully avoids an impedance mismatch between the data returned and the
required updates to the datastore.

This does not replicate all the database tables as many of the database tables
are indexes that can be recovered from the contents of other tables.

The format and parameters for the api are documented in
``synapse/replication/resource.py``.
The Replication Protocol
~~~~~~~~~~~~~~~~~~~~~~~~

See ``tcp_replication.rst``


The Slaved DataStore
Expand Down
15 changes: 5 additions & 10 deletions docs/workers.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ across multiple processes is a recipe for disaster, plus you should be using
postgres anyway if you care about scalability).

The workers communicate with the master synapse process via a synapse-specific
HTTP protocol called 'replication' - analogous to MySQL or Postgres style
TCP protocol called 'replication' - analogous to MySQL or Postgres style
database replication; feeding a stream of relevant data to the workers so they
can be kept in sync with the main synapse process and database state.

Expand All @@ -21,16 +21,11 @@ To enable workers, you need to add a replication listener to the master synapse,
listeners:
- port: 9092
bind_address: '127.0.0.1'
type: http
tls: false
x_forwarded: false
resources:
- names: [replication]
compress: false
type: replication

Under **no circumstances** should this replication API listener be exposed to the
public internet; it currently implements no authentication whatsoever and is
unencrypted HTTP.
unencrypted.

You then create a set of configs for the various worker processes. These should be
worker configuration files should be stored in a dedicated subdirectory, to allow
Expand All @@ -57,7 +52,8 @@ For instance::
worker_app: synapse.app.synchrotron

# The replication listener on the synapse to talk to.
worker_replication_url: http://127.0.0.1:9092/_synapse/replication
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is still a reference to worker_replication_url in the previous para

worker_replication_host: 127.0.0.1
worker_replication_port: 9092

worker_listeners:
- type: http
Expand Down Expand Up @@ -95,4 +91,3 @@ To manipulate a specific worker, you pass the -w option to synctl::
All of the above is highly experimental and subject to change as Synapse evolves,
but documenting it here to help folks needing highly scalable Synapses similar
to the one running matrix.org!