Reduce confusion between topological and stream position tokens (SPEC-268)

[matrixbot]

The client-server API has two types of position identifiers, representing two views of the ordering of events:

• topological (tNN-NNN_NNN_NNN_NNN) (aka pagination token)
• stream (sNNN_NNN_NNN_NNN)

A topological position only makes sense within the context of a particular room, and refers to a particular point on the event DAG. The events between two topo tokens are subject to change, as the DAG changes over time.

A stream position is a global identifier across all rooms, and refers to a point on the timeline of when the server received the event (which may be wildly different to where it fits on the graph, or where it happened in a different server's timeline).

/sync returns events in the stream timeline order, and takes a stream position token as a 'since' parameter.

/room/<x>/messages works in the topological timeline order, and takes a topological token as a 'from' parameter.

It's easy to misunderstand what one can do with these tokens. In particular, /sync currently returns:

• next_batch: a streaming token from which streaming can be resumed next time
• rooms.x.timeline.prev_batch: a topological token. The idea is that you can back-populate from here.

We should do our best to reduce the confusion by making sure that we have a consistent set of names which make it clear what sort of token is expected.

(Imported from https://matrix.org/jira/browse/SPEC-268)

(Reported by @richvdh)

[matrixbot]

Jira watchers: @richvdh

[matrixbot]

I floated one idea, which was to make /messages take both sorts of tokens, and don't make clients worry about the differences.

However, on further consideration, even if we did allow /messages to take both sorts of tokens, I think a client really needs to understand the difference between the two sorts of token (for instance: the scoping to room vs global). [~erikj] thought it would probably only add to the confusion. So I think that idea is probably flawed.

-- @richvdh

[matrixbot]

So should we just rename the next_batch field in the /sync response? How about global_stream_position? (I think 'next' can be implicit here, since the stream position explicitly only ever goes forward.)

-- @richvdh

[matrixbot]

A decision on this has been deferred pending figuring out where we want to go with topo tokens and stream tokens in general (and whether it makes sense to support topo tokens in the C-S API at all)

-- @richvdh

[bwindels]

Also, at least synapse accepts a stream position token in /messages?from=, but this is not mentioned in the spec.