From 3c50306d181fa90d7c8e96c63d593a88a92e408a Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Tue, 4 Jul 2023 15:29:03 +0100 Subject: [PATCH 01/51] Initial draft --- proposals/XXXX-event-thread-and-order.md | 209 +++++++++++++++++++++++ 1 file changed, 209 insertions(+) create mode 100644 proposals/XXXX-event-thread-and-order.md diff --git a/proposals/XXXX-event-thread-and-order.md b/proposals/XXXX-event-thread-and-order.md new file mode 100644 index 00000000000..1194a45a439 --- /dev/null +++ b/proposals/XXXX-event-thread-and-order.md @@ -0,0 +1,209 @@ +# MSCXXXX: Providing thread and order for all events to allow consistent read receipt handling + +We argue that we have made it unnecessarily hard for clients and servers to +decide whether a message is read or unread, and we can solve this problem by +clarifying the definitions, and including extra pieces of information with +every event that help to identify its thread(s) and ordering. + +## Motivation + +In order to decide whether a room is unread, a Matrix client must decide whether +it contains any unread messages. + +In order to decide whether a room has notifications, a Matrix client or server +must decide whether any of its potentially-notifying messages is unread. + +Both of these tasks require us to decide whether a message is read or unread. + +To make this decision we have receipts. There are two types of receipt: threaded +and unthreaded. + +To decide whether an event is read, we use the following rule: + +> An event is read if the room contains an unthreaded receipt pointing at an +> event which is *after or the same as* the event, or a threaded receipt +> pointing at an event that is *in the same thread* as the event, and is *after +> or the same as* the event. +> +> Otherwise, it is unread. + +(In both cases we only consider receipts sent by the current user, obviously. We +consider either private or public read receipts.) + +We do not propose to change the above definition. + +To perform this calculation we need definitions of *after or the same as* and *in the same +thread*. + +The meaning *the same as* is clear: if a receipt points at an event, then that +event is read. + +### Current definition of *after* + +The current spec (see +[11.6](https://spec.matrix.org/latest/client-server-api/#receipts) is not clear +about what it calls "read up to" means. + +Clients like Element Web make the assumption that *after* means "after in Sync +Order", where "Sync Order" means "the order in which I (the client) received the +events from the server via sync", so if a client received an event and another +event for which it has a receipt via sync, then the event that was later in the +sync or received in a later sync, is after the other one. We think this is +similar to Stream Ordering, which is mentioned once in the spec without +defintion [7.6 +Syncing](https://spec.matrix.org/unstable/client-server-api/#syncing), but we +are not certain that it is identical, because we believe it may be possible for +different clients to receive events in a different order from each other for the +same account. + +See also [Spec Issue #1167](https://github.com/matrix-org/matrix-spec/issues/1167). + +### Current definition of *in the same thread* + +See [11.6.22 Threaded read +receipts](https://spec.matrix.org/latest/client-server-api/#threaded-read-receipts), +but a summary in looser language is: + +An event is in thread A if: + +* its event ID is A and it has `m.thread` children (i.e. it is the thread root), or +* it has an `m.thread` relationship to the event with ID A (i.e. it is a message + in the thread), or +* it has an ancestor event (found by traversing relationships) that fits either of the above. + +An event is in the main thread if: + +* it is not in another thread, or +* its ancestor events do not include an `m.thread` relationship + +Note that thread root events, and their non-threaded children (e.g. reactions to +thread root events) are in BOTH the main thread and another thread. + +### Problems with the current definitions + +The current definition of *after* is ambiguous, and difficult for clients to +calculate. It depends on only receiving events via sync, which is impractical +due to backpaginating, and the desire to fetch thread messages via the +`relations` API. + +Further, we believe that the current working definition of "Sync Order" does not +make sense, because different clients may receive events in different orders, +meaning that a receipt from one client would be interpreted differently by +another client. + +The current definition of *in the same thread* is difficult for clients to +calculate because it requires fetching all parents of an event to identify which +thread it belongs to. In the meantime, clients must deal with events and +receipts that are "homeless". + +The current definitions also make it needlessly complex for clients to determine +whether an event is read because they must fetch the event that is referred to by +a receipt before they can make the decision, and hold the receipt "in limbo" +before it is available. + +## Proposal + +We propose to tighten up the definitions, and include extra information in the +events provided by the server to make it easy to calculate whether an event is +read. + +### Proposed definition of *after* + +We propose that the definiton of *after* should be: + +* Event A is after event B if its Stream Order is larger. +* Where A and B have the same Stream Order, A is after B if A's event ID is + lexicographically after B's event ID. + +### Proposed definition of *in the same thread* + +We propose that the definition of *in the same thread* should use this wording: + +An event is in the `main` thread if: + +* it has no `m.thread_id` property + +An event is in a non-main thread if: + +* it has an `m.thread_id` property (i.e. it is in the thread whose ID matches + the value of this property), or +* it contains an `m.is_thread_root` property with value `true`, then it is in + the thread whose ID is its event ID (i.e. it is a thread root). + +Note: thread roots are in TWO threads: the one whose thread ID equals their +event ID, AND the `main` thread. + +Note: other events with relationships to the thread root (e.g. reactions to the +thread root) are NOT considered part of the thread. This is a change to the +current definition. We propose that threads should not be considered unread if a +new reaction to the thread root is received. + +### Supporting changes to event structure + +We propose: + +* all events in a thread should contain an `m.thread_id` property. +* all events should contain an `m.stream_order` property. +* all thread root events should contain an `m.is_thread_root` property with + value `true`. +* all receipts should contain an `m.stream_order` property alongside `m.read` + and/or `m.read.private` inside the information about an event, which is a + cache of the `m.stream_order` property within the referred-to event. + +This makes it explicit how the server has categorised events, meaning clients +and servers can always agree on what is a thread root, a threaded message, or a +main thread message. + +It also makes it explicit which event is before or after another event in Stream +Order. + +This prevents disagreements between clients and servers about the meaning of a +particular read receipt, and by including the Stream Order of the referred event +in the receipt, avoids the need to fetch that event in order to make a decision +about which events are read. + +## Potential issues + +This special-cases threads over other relationships. + +But, this was already the case with threaded receipts, and we believe it is +necessary to provide consistent behaviour. + +The change to consider reactions to thread roots as outside of the thread may be +inconvenient for clients, because they will probably want to display those +reactions in any "thread view" they display. We consider this inconvenience +worthwhile because it is necessary to ensure the semantics of read receipts make +sense. We do not think that reactions to a thread root should mark that thread +as unread (but we possibly could be persuaded?). + +## Alternatives + +This proposal would replace +[MSC4023: Thread ID for 2nd order-relation](https://github.com/matrix-org/matrix-spec-proposals/pull/4023) +and the idea that +[MSC3051: A scalable relation format](https://github.com/matrix-org/matrix-spec-proposals/pull/3051) +would help fix receipts. + +This propsal would not replace +[MSC3981: /relations recursion](https://github.com/matrix-org/matrix-spec-proposals/pull/3981) +but would make it less important, because we would no longer depend on the +server providing messages in Sync Order, so we could happily fetch messages +recursively and still be able to slot them into the right thread and ordering. + +Note that the expectation (from some client devs e.g. me @andybalaam) was that +MSC3981 would solve many problems for clients because the events in a thread +would be returned in Sync Order, but this is not true: the proposal will return +events in Topological Order, which is useless for determining which events are +read. + +## Security considerations + +None highlighted so far. + +## Unstable prefix + +TODO + +## Dependencies + +None at this time. From e7983f2bb7b86b8f904bc66673611e44efd72484 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Tue, 4 Jul 2023 15:31:08 +0100 Subject: [PATCH 02/51] Update number to MSC4033 --- ...event-thread-and-order.md => 4033-event-thread-and-order.md} | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) rename proposals/{XXXX-event-thread-and-order.md => 4033-event-thread-and-order.md} (99%) diff --git a/proposals/XXXX-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md similarity index 99% rename from proposals/XXXX-event-thread-and-order.md rename to proposals/4033-event-thread-and-order.md index 1194a45a439..ec0220bab71 100644 --- a/proposals/XXXX-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -1,4 +1,4 @@ -# MSCXXXX: Providing thread and order for all events to allow consistent read receipt handling +# MSC4033: Providing thread and order for all events to allow consistent read receipt handling We argue that we have made it unnecessarily hard for clients and servers to decide whether a message is read or unread, and we can solve this problem by From a75a8e58e8f7e64a2cc357dc448f41c29143dd2a Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Tue, 4 Jul 2023 15:33:58 +0100 Subject: [PATCH 03/51] Fix mis-spelling --- proposals/4033-event-thread-and-order.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index ec0220bab71..681621cb4c1 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -50,7 +50,7 @@ events from the server via sync", so if a client received an event and another event for which it has a receipt via sync, then the event that was later in the sync or received in a later sync, is after the other one. We think this is similar to Stream Ordering, which is mentioned once in the spec without -defintion [7.6 +definition [7.6 Syncing](https://spec.matrix.org/unstable/client-server-api/#syncing), but we are not certain that it is identical, because we believe it may be possible for different clients to receive events in a different order from each other for the From 6841e75de0283ac237e0543c33e08ad853fd9dcd Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Tue, 4 Jul 2023 15:38:12 +0100 Subject: [PATCH 04/51] Fix another mis-spelling --- proposals/4033-event-thread-and-order.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 681621cb4c1..52fd029bf22 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -109,7 +109,7 @@ read. ### Proposed definition of *after* -We propose that the definiton of *after* should be: +We propose that the definition of *after* should be: * Event A is after event B if its Stream Order is larger. * Where A and B have the same Stream Order, A is after B if A's event ID is From 2521212e540647ce99ee1931cc007fcba237a7f2 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Tue, 4 Jul 2023 15:47:25 +0100 Subject: [PATCH 05/51] Add some more alternatives --- proposals/4033-event-thread-and-order.md | 25 ++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 52fd029bf22..a432201623f 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -112,8 +112,6 @@ read. We propose that the definition of *after* should be: * Event A is after event B if its Stream Order is larger. -* Where A and B have the same Stream Order, A is after B if A's event ID is - lexicographically after B's event ID. ### Proposed definition of *in the same thread* @@ -178,6 +176,8 @@ as unread (but we possibly could be persuaded?). ## Alternatives +### This replaces other attempts to fix receipts + This proposal would replace [MSC4023: Thread ID for 2nd order-relation](https://github.com/matrix-org/matrix-spec-proposals/pull/4023) and the idea that @@ -196,6 +196,27 @@ would be returned in Sync Order, but this is not true: the proposal will return events in Topological Order, which is useless for determining which events are read. +### Avoiding saying "Stream Order" + +We could avoid the phrase Stream Order in this proposal, and instead simply talk +about a consistent order that the server and client agree on because it is +included with event info. + +We could even note that we expect it to be Stream Order for homeservers that +have such a concept, but the important thing for us is that it is consistent and +explicit. + +### The server calculates unread status + +We could use the definitions within this proposal but avoid calculating what was +unread on the client. Instead we could ask the server to figure out which rooms +are unread. + +The client will still need to know which events are unread in order to process +notifications that are encrypted when they pass through the server, so this +proposal would probably be unaltered even if we added the capability for servers +to surface which rooms are unread. + ## Security considerations None highlighted so far. From fe83ad991ecf76eaf787003ada8219a5d95e6e43 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Tue, 4 Jul 2023 15:48:17 +0100 Subject: [PATCH 06/51] Add TODO list --- proposals/4033-event-thread-and-order.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index a432201623f..859fdc3c5f9 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -1,5 +1,21 @@ # MSC4033: Providing thread and order for all events to allow consistent read receipt handling +## TODO + +- [x] review request Patrick +- [x] Mention no need for the words Stream Order to be included +- [x] Mention server-side calculation of unreadness +- [ ] Acknowledge patrick +- [ ] Move ordering into unsigned +- [ ] Events can never have the same stream order +- [ ] Reword to some generic order instead of stream order? +- [ ] Can't do m.is_thread_root because we don't know until a child exists +- [ ] Example of inconsistent Sync Order: +- [ ] I guess that happens if one is doing an incremental sync and one is doing an initial sync and the event in question is the latest event by stream order but a much earlier event by topo order? +- [ ] Explicitly say we should update the spec wording around what we mean by read-up-to +- [ ] Consider the thread root not being in the thread. Would need to think about whether it matters if the thread root is somehow later than a thread message in Stream Order. +- [ ] m.thread_id should be preserved through a redaction. Note that this will need a new room version + We argue that we have made it unnecessarily hard for clients and servers to decide whether a message is read or unread, and we can solve this problem by clarifying the definitions, and including extra pieces of information with From 3ae4995eb554e22d2c40c565a672b87e675b4373 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Tue, 4 Jul 2023 15:49:45 +0100 Subject: [PATCH 07/51] Add missing bracket --- proposals/4033-event-thread-and-order.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 859fdc3c5f9..c5ace88f1f0 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -57,7 +57,7 @@ event is read. ### Current definition of *after* The current spec (see -[11.6](https://spec.matrix.org/latest/client-server-api/#receipts) is not clear +[11.6 Receipts](https://spec.matrix.org/latest/client-server-api/#receipts)) is not clear about what it calls "read up to" means. Clients like Element Web make the assumption that *after* means "after in Sync From 8414c2eb8b0fe120de6785acadd3ddeebbef3632 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Tue, 4 Jul 2023 17:09:27 +0100 Subject: [PATCH 08/51] Add an acknowledgements section --- proposals/4033-event-thread-and-order.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index c5ace88f1f0..a4d46829257 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -5,7 +5,7 @@ - [x] review request Patrick - [x] Mention no need for the words Stream Order to be included - [x] Mention server-side calculation of unreadness -- [ ] Acknowledge patrick +- [x] Acknowledge patrick and others - [ ] Move ordering into unsigned - [ ] Events can never have the same stream order - [ ] Reword to some generic order instead of stream order? @@ -244,3 +244,8 @@ TODO ## Dependencies None at this time. + +## Acknowledgements + +Formed from a discussion with @ara4n, with early review from @clokep. Built on +ideas from @t3chguy, @justjanne, @germain-gg and @weeman1337. From 7e26924de9fe76a1d361a8e5f0eb07f4d7dd7115 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 08:32:16 +0100 Subject: [PATCH 09/51] Make thread roots not in their thread --- proposals/4033-event-thread-and-order.md | 54 ++++++++++++------------ 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index a4d46829257..750b406aa56 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -6,15 +6,17 @@ - [x] Mention no need for the words Stream Order to be included - [x] Mention server-side calculation of unreadness - [x] Acknowledge patrick and others -- [ ] Move ordering into unsigned +- [ ] Make it robust to redactions by always considering redacted messages read +- [x] Remove thread roots from the thread +- [x] ~~Move ordering into unsigned (maybe thread id too?)~~ No - unsigned is frowned upon - [ ] Events can never have the same stream order - [ ] Reword to some generic order instead of stream order? -- [ ] Can't do m.is_thread_root because we don't know until a child exists +- [x] Can't do m.is_thread_root because we don't know until a child exists - [ ] Example of inconsistent Sync Order: - [ ] I guess that happens if one is doing an incremental sync and one is doing an initial sync and the event in question is the latest event by stream order but a much earlier event by topo order? - [ ] Explicitly say we should update the spec wording around what we mean by read-up-to - [ ] Consider the thread root not being in the thread. Would need to think about whether it matters if the thread root is somehow later than a thread message in Stream Order. -- [ ] m.thread_id should be preserved through a redaction. Note that this will need a new room version +- [ ] m.thread_id should be preserved through a redaction. Note that this will need a new room version Would moving into unsigned prevent this? We argue that we have made it unnecessarily hard for clients and servers to decide whether a message is read or unread, and we can solve this problem by @@ -133,24 +135,18 @@ We propose that the definition of *after* should be: We propose that the definition of *in the same thread* should use this wording: -An event is in the `main` thread if: +An event is in a non-main-thread if: -* it has no `m.thread_id` property +* it has an `m.thread_id` property -An event is in a non-main thread if: +Otherwise, it is in the `main` thread. -* it has an `m.thread_id` property (i.e. it is in the thread whose ID matches - the value of this property), or -* it contains an `m.is_thread_root` property with value `true`, then it is in - the thread whose ID is its event ID (i.e. it is a thread root). +No events are in more than one thread. -Note: thread roots are in TWO threads: the one whose thread ID equals their -event ID, AND the `main` thread. - -Note: other events with relationships to the thread root (e.g. reactions to the -thread root) are NOT considered part of the thread. This is a change to the -current definition. We propose that threads should not be considered unread if a -new reaction to the thread root is received. +Note: this means that thread roots are in the `main` thread, and not in the +thread branching from them. Non-thread children of thread roots (e.g. +reactions to a thread root) are also in the `main` thread. This is a change to +the current definition. ### Supporting changes to event structure @@ -158,15 +154,18 @@ We propose: * all events in a thread should contain an `m.thread_id` property. * all events should contain an `m.stream_order` property. -* all thread root events should contain an `m.is_thread_root` property with - value `true`. * all receipts should contain an `m.stream_order` property alongside `m.read` and/or `m.read.private` inside the information about an event, which is a cache of the `m.stream_order` property within the referred-to event. +The server should include an `m.thread_id` property in any event that has an +ancestor relationship that includes an `m.thread` relationship. The value of +`m.thread_id` is the event ID of the event referenced by the `m.thread` +relationship. + This makes it explicit how the server has categorised events, meaning clients -and servers can always agree on what is a thread root, a threaded message, or a -main thread message. +and servers can always agree on which thread an event is in, so the meaning of +threaded receipt is clear. It also makes it explicit which event is before or after another event in Stream Order. @@ -183,12 +182,13 @@ This special-cases threads over other relationships. But, this was already the case with threaded receipts, and we believe it is necessary to provide consistent behaviour. -The change to consider reactions to thread roots as outside of the thread may be -inconvenient for clients, because they will probably want to display those -reactions in any "thread view" they display. We consider this inconvenience -worthwhile because it is necessary to ensure the semantics of read receipts make -sense. We do not think that reactions to a thread root should mark that thread -as unread (but we possibly could be persuaded?). +The change to consider thread roots (and reactions to them) as outside of the +thread may be inconvenient for clients, because they will probably want to +display those events in any "thread view" they display. We consider this +inconvenience worthwhile because it is necessary to ensure the semantics of read +receipts make sense. We do not think that reactions or edits to a thread root +should mark that thread as unread - instead they mark the main thread as unread, +which the client can use to draw attention to the thread root. ## Alternatives From ee19047dc57f1e992d2e43125a16306b0908cc5f Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 08:33:50 +0100 Subject: [PATCH 10/51] Add a changelog --- proposals/4033-event-thread-and-order.md | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 750b406aa56..788d26198a1 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -249,3 +249,8 @@ None at this time. Formed from a discussion with @ara4n, with early review from @clokep. Built on ideas from @t3chguy, @justjanne, @germain-gg and @weeman1337. + +## Changelog + +* 2023-07-04 Initial draft by @andybalaam after conversation with @ara4n. +* 2023-07-05 Remove thread roots from their thread after conversation with @clokep. From 0e17420952fd5af76e8ec0c6dc57d93ecf30fea8 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 10:42:09 +0100 Subject: [PATCH 11/51] Clarify stream order and handle redactions --- proposals/4033-event-thread-and-order.md | 84 ++++++++++++++++++++---- 1 file changed, 72 insertions(+), 12 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 788d26198a1..33726526f84 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -6,17 +6,18 @@ - [x] Mention no need for the words Stream Order to be included - [x] Mention server-side calculation of unreadness - [x] Acknowledge patrick and others -- [ ] Make it robust to redactions by always considering redacted messages read +- [x] Make it robust to redactions by always considering redacted messages read - [x] Remove thread roots from the thread - [x] ~~Move ordering into unsigned (maybe thread id too?)~~ No - unsigned is frowned upon - [ ] Events can never have the same stream order -- [ ] Reword to some generic order instead of stream order? +- [x] Reword to some generic order instead of stream order? - [x] Can't do m.is_thread_root because we don't know until a child exists - [ ] Example of inconsistent Sync Order: - [ ] I guess that happens if one is doing an incremental sync and one is doing an initial sync and the event in question is the latest event by stream order but a much earlier event by topo order? - [ ] Explicitly say we should update the spec wording around what we mean by read-up-to -- [ ] Consider the thread root not being in the thread. Would need to think about whether it matters if the thread root is somehow later than a thread message in Stream Order. -- [ ] m.thread_id should be preserved through a redaction. Note that this will need a new room version Would moving into unsigned prevent this? +- [x] Consider the thread root not being in the thread. Would need to think about whether it matters if the thread root is somehow later than a thread message in Stream Order. +- [ ] Be really clear on the JSON changes, and the semantics changes +- [ ] Suggest that `ts` in receipts might be redundant? We argue that we have made it unnecessarily hard for clients and servers to decide whether a message is read or unread, and we can solve this problem by @@ -101,7 +102,7 @@ thread root events) are in BOTH the main thread and another thread. The current definition of *after* is ambiguous, and difficult for clients to calculate. It depends on only receiving events via sync, which is impractical -due to backpaginating, and the desire to fetch thread messages via the +due to backpagination, and the desire to fetch thread messages via the `relations` API. Further, we believe that the current working definition of "Sync Order" does not @@ -121,15 +122,57 @@ before it is available. ## Proposal -We propose to tighten up the definitions, and include extra information in the -events provided by the server to make it easy to calculate whether an event is -read. +We propose to modify the definitions slightly, and include extra information in +the events provided by the server to make it easy to calculate whether an event +is read. ### Proposed definition of *after* We propose that the definition of *after* should be: -* Event A is after event B if its Stream Order is larger. +* Event A is after event B if its *Stream Order* is larger. + +We define *Stream Order* to be an immutable, unique number attached to an event +on creation that defines the order of events in regard to receipts. + +Note: some home servers already have a concept of Stream Order, and we intend +that this definition is consistent with that as currently implemented. But, for +the purposes of this proposal the only important aspect of Stream Order is that +server and clients agree that receipts apply to events with a lower Stream +Order, and that the Stream Order of an event never changes. + +If the name Stream Order proves confusing, this proposal can function equally +well using a different name: it simply needs to be unique, immutable, and +increase for "newer" messages. + +We do not require that Stream Order be consistent across federation (in fact, we +believe that this would be impossible to achieve). The only requirement is that +all the clients for a user and the one server to which they connect agree. For +this reason, we propose that `m.order` be included in an event's `unsigned` +property. + +### Notes on the meaning of Stream Order + +Because it controls the meaning of read receipts, it is desirable that Stream +Order be as close as possible to Sync Order, the order in which clients receive +events via sync. However, since clients receive events in different orders +depending on the APIs they use, it is not a goal that Stream Order exactly match +the order in which clients receive events. Instead, it provides a canonical +order that means we can be clear about what the user has read, and thus should +generally increase for "newer" messages. Clients may decide to re-order events +into Stream Order, or they may decide to display unread messages higher up the +timeline if the orders do not match the order they choose for display. + +Because Stream Order may be inconsistent across federation (and, in theory, across +different users on the same home server, although we expect in practice this +will not happen), one user may occasionally see a different unread status for +another user from what that user themselves see. We regard this as impossible to +avoid, and expect that in most cases it will be unnoticeable, since home servers +with good connectivity will normally have similar Stream Order. When servers have +long network splits, there will be a noticeable difference at first, but once +messages start flowing normally and users start reading them, the differences +will disappear as new events will have higher Stream order than the older ones +on both servers. ### Proposed definition of *in the same thread* @@ -148,15 +191,28 @@ thread branching from them. Non-thread children of thread roots (e.g. reactions to a thread root) are also in the `main` thread. This is a change to the current definition. +### Proposed change in consideration of redacted events + +We propose that redacted events and redaction events should never be considered +unread. + +This avoids the need to identify which thread a redacted event belongs to, which +will be difficult if its `m.thread_id` property has been stripped out. + +Since we propose that receipts contain the `m.order` of their referred-to event, +this means we do not need to look within a redacted event for its Stream Order, +because the receipt provides it. This avoids needing to preserve the `m.order` +property when redacting events. + ### Supporting changes to event structure We propose: * all events in a thread should contain an `m.thread_id` property. -* all events should contain an `m.stream_order` property. -* all receipts should contain an `m.stream_order` property alongside `m.read` +* all events should contain an `m.order` property. +* all receipts should contain an `m.order` property alongside `m.read` and/or `m.read.private` inside the information about an event, which is a - cache of the `m.stream_order` property within the referred-to event. + cache of the `m.order` property within the referred-to event. The server should include an `m.thread_id` property in any event that has an ancestor relationship that includes an `m.thread` relationship. The value of @@ -254,3 +310,7 @@ ideas from @t3chguy, @justjanne, @germain-gg and @weeman1337. * 2023-07-04 Initial draft by @andybalaam after conversation with @ara4n. * 2023-07-05 Remove thread roots from their thread after conversation with @clokep. +* 2023-07-05 Handle redactions after conversation with @t3chguy +* 2023-07-05 Give a definition of Stream Order, caveat around its name +* 2023-07-05 Be explicit about Stream Order not going over federation +* 2023-07-05 Mention disagreeing about what another user has read From a32a3f9b8476680879a7a6024c55e455475cb11d Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 10:54:18 +0100 Subject: [PATCH 12/51] Small clarifications --- proposals/4033-event-thread-and-order.md | 67 ++++++++++++++---------- 1 file changed, 40 insertions(+), 27 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 33726526f84..d4852c18e18 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -9,15 +9,14 @@ - [x] Make it robust to redactions by always considering redacted messages read - [x] Remove thread roots from the thread - [x] ~~Move ordering into unsigned (maybe thread id too?)~~ No - unsigned is frowned upon -- [ ] Events can never have the same stream order +- [x] Events can never have the same stream order - [x] Reword to some generic order instead of stream order? - [x] Can't do m.is_thread_root because we don't know until a child exists -- [ ] Example of inconsistent Sync Order: -- [ ] I guess that happens if one is doing an incremental sync and one is doing an initial sync and the event in question is the latest event by stream order but a much earlier event by topo order? -- [ ] Explicitly say we should update the spec wording around what we mean by read-up-to +- [x] Example of inconsistent Sync Order +- [x] Explicitly say we should update the spec wording around what we mean by read-up-to - [x] Consider the thread root not being in the thread. Would need to think about whether it matters if the thread root is somehow later than a thread message in Stream Order. -- [ ] Be really clear on the JSON changes, and the semantics changes -- [ ] Suggest that `ts` in receipts might be redundant? +- [ ] JSON examples +- [x] Suggest that `ts` in receipts might be redundant? We argue that we have made it unnecessarily hard for clients and servers to decide whether a message is read or unread, and we can solve this problem by @@ -67,13 +66,17 @@ Clients like Element Web make the assumption that *after* means "after in Sync Order", where "Sync Order" means "the order in which I (the client) received the events from the server via sync", so if a client received an event and another event for which it has a receipt via sync, then the event that was later in the -sync or received in a later sync, is after the other one. We think this is -similar to Stream Ordering, which is mentioned once in the spec without -definition [7.6 -Syncing](https://spec.matrix.org/unstable/client-server-api/#syncing), but we -are not certain that it is identical, because we believe it may be possible for -different clients to receive events in a different order from each other for the -same account. +sync or received in a later sync, is after the other one [^1]. + +[^1]: We think this is similar to Stream Ordering, which is mentioned once in + the spec without definition + [7.6 Syncing](https://spec.matrix.org/unstable/client-server-api/#syncing), + but we are not certain that it is identical, because we believe it may be + possible for different clients to receive events in a different order from + each other for the same account. For example, if one client is doing an + incremental sync, and another is doing an initial sync, recently-arrived + events that are "old "in Topological Order may be received in different orders + on the two clients. See also [Spec Issue #1167](https://github.com/matrix-org/matrix-spec/issues/1167). @@ -135,7 +138,13 @@ We propose that the definition of *after* should be: We define *Stream Order* to be an immutable, unique number attached to an event on creation that defines the order of events in regard to receipts. -Note: some home servers already have a concept of Stream Order, and we intend +We propose updating the spec around receipts +([11.6 Receipts](https://spec.matrix.org/latest/client-server-api/#receipts)) +to be explicit about what "read up to" means, using the above definition. + +#### Notes + +Some home servers already have a concept of Stream Order, and we intend that this definition is consistent with that as currently implemented. But, for the purposes of this proposal the only important aspect of Stream Order is that server and clients agree that receipts apply to events with a lower Stream @@ -163,16 +172,17 @@ generally increase for "newer" messages. Clients may decide to re-order events into Stream Order, or they may decide to display unread messages higher up the timeline if the orders do not match the order they choose for display. -Because Stream Order may be inconsistent across federation (and, in theory, across -different users on the same home server, although we expect in practice this -will not happen), one user may occasionally see a different unread status for -another user from what that user themselves see. We regard this as impossible to -avoid, and expect that in most cases it will be unnoticeable, since home servers -with good connectivity will normally have similar Stream Order. When servers have -long network splits, there will be a noticeable difference at first, but once -messages start flowing normally and users start reading them, the differences -will disappear as new events will have higher Stream order than the older ones -on both servers. +Because Stream Order may be inconsistent across federation[^2], one user may +occasionally see a different unread status for another user from what that user +themselves see. We regard this as impossible to avoid, and expect that in most +cases it will be unnoticeable, since home servers with good connectivity will +normally have similar Stream Order. When servers have long network splits, there +will be a noticeable difference at first, but once messages start flowing +normally and users start reading them, the differences will disappear as new +events will have higher Stream order than the older ones on both servers. + +[^2]: In theory, Stream Order could also be inconsistent across different users + on the same home server, although we expect in practice this will not happen. ### Proposed definition of *in the same thread* @@ -212,7 +222,10 @@ We propose: * all events should contain an `m.order` property. * all receipts should contain an `m.order` property alongside `m.read` and/or `m.read.private` inside the information about an event, which is a - cache of the `m.order` property within the referred-to event. + cache of the `m.order` property within the referred-to event [^3]. + +[^3]: This might make the `ts` property within receipts redundant. We are not + actually sure what purpose this property is intended to serve. The server should include an `m.thread_id` property in any event that has an ancestor relationship that includes an `m.thread` relationship. The value of @@ -310,7 +323,7 @@ ideas from @t3chguy, @justjanne, @germain-gg and @weeman1337. * 2023-07-04 Initial draft by @andybalaam after conversation with @ara4n. * 2023-07-05 Remove thread roots from their thread after conversation with @clokep. -* 2023-07-05 Handle redactions after conversation with @t3chguy -* 2023-07-05 Give a definition of Stream Order, caveat around its name +* 2023-07-05 Make redactions never unread after conversation with @t3chguy +* 2023-07-05 Give a definition of Stream Order * 2023-07-05 Be explicit about Stream Order not going over federation * 2023-07-05 Mention disagreeing about what another user has read From 58102cfcccf59e6f88e97f87f6b1c55538b75ea6 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 13:26:04 +0100 Subject: [PATCH 13/51] Include updated defintion of read/unread event --- proposals/4033-event-thread-and-order.md | 25 ++++++++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index d4852c18e18..87db2f0231d 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -28,8 +28,9 @@ every event that help to identify its thread(s) and ordering. In order to decide whether a room is unread, a Matrix client must decide whether it contains any unread messages. -In order to decide whether a room has notifications, a Matrix client or server -must decide whether any of its potentially-notifying messages is unread. +Similarly, in order to decide whether a room has notifications, a Matrix client +or server must decide whether any of its potentially-notifying messages is +unread. Both of these tasks require us to decide whether a message is read or unread. @@ -244,6 +245,26 @@ particular read receipt, and by including the Stream Order of the referred event in the receipt, avoids the need to fetch that event in order to make a decision about which events are read. +### Definition of read and unread events + +We propose that the definition of whether an event is read should be: + +> An event is read if the room contains an unthreaded receipt pointing at an +> event which is *after or the same as* the event, or a threaded receipt +> pointing at an event that is *in the same thread* as the event, and is *after +> or the same as* the event. +> +> Because the receipt itself contains the `m.order` of the pointed-to event, +> there is no need to examine the pointed-to event, so it is sufficient to +> compare the `m.order` of the event in question with the `m.order` in the +> receipt. +> +> Redacted events are always read. +> +> Otherwise, it is unread. + +Obviously, this definition depends on the definitions above. + ## Potential issues This special-cases threads over other relationships. From 8ecb0bbab9f20892b7e879f5657875a9704f03d4 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 13:44:16 +0100 Subject: [PATCH 14/51] Provide JSON examples --- proposals/4033-event-thread-and-order.md | 78 ++++++++++++++++++------ 1 file changed, 61 insertions(+), 17 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 87db2f0231d..537eac612d1 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -15,7 +15,7 @@ - [x] Example of inconsistent Sync Order - [x] Explicitly say we should update the spec wording around what we mean by read-up-to - [x] Consider the thread root not being in the thread. Would need to think about whether it matters if the thread root is somehow later than a thread message in Stream Order. -- [ ] JSON examples +- [x] JSON examples - [x] Suggest that `ts` in receipts might be redundant? We argue that we have made it unnecessarily hard for clients and servers to @@ -158,7 +158,7 @@ increase for "newer" messages. We do not require that Stream Order be consistent across federation (in fact, we believe that this would be impossible to achieve). The only requirement is that all the clients for a user and the one server to which they connect agree. For -this reason, we propose that `m.order` be included in an event's `unsigned` +this reason, we propose that `order` be included in an event's `unsigned` property. ### Notes on the meaning of Stream Order @@ -191,7 +191,7 @@ We propose that the definition of *in the same thread* should use this wording: An event is in a non-main-thread if: -* it has an `m.thread_id` property +* it has a `thread_id` property Otherwise, it is in the `main` thread. @@ -208,29 +208,71 @@ We propose that redacted events and redaction events should never be considered unread. This avoids the need to identify which thread a redacted event belongs to, which -will be difficult if its `m.thread_id` property has been stripped out. +will be difficult if its `thread_id` property has been stripped out. -Since we propose that receipts contain the `m.order` of their referred-to event, +Since we propose that receipts contain the `order` of their referred-to event, this means we do not need to look within a redacted event for its Stream Order, -because the receipt provides it. This avoids needing to preserve the `m.order` +because the receipt provides it. This avoids needing to preserve the `order` property when redacting events. ### Supporting changes to event structure +Example event (changes are highlighted in bold): + +{ + "content": { + "body": "This is an example text message", + "format": "org.matrix.custom.html", + "formatted_body": "<b>This is an example text message</b>", + "msgtype": "m.text" + }, + "event_id": "$143273582443PhrSn:example.org", + "origin_server_ts": 1432735824653, + "room_id": "!jEsUZKDJdhlrceRyVU:example.org", + "thread_id": "$fgsUZKDJdhlrceRyhj", // Optional + "sender": "@example:example.org", + "type": "m.room.message", + "unsigned": { + "age": 1234, + "order": 56764334543 + } +} + +Example receipt (changes are highlighted in bold): + +{ + "content": { + "$1435641916114394fHBLK:matrix.org": { + "order": 56764334544, + "m.read": { + "@rikj:jki.re": { + "ts": 1436451550453 + } + }, + "m.read.private": { + "@self:example.org": { + "ts": 1661384801651 + } + } + } + }, + "type": "m.receipt" +} + We propose: -* all events in a thread should contain an `m.thread_id` property. -* all events should contain an `m.order` property. -* all receipts should contain an `m.order` property alongside `m.read` +* all events in a thread should contain a `thread_id` property. +* all events should contain an `order` property. +* all receipts should contain an `order` property alongside `m.read` and/or `m.read.private` inside the information about an event, which is a - cache of the `m.order` property within the referred-to event [^3]. + cache of the `order` property within the referred-to event [^3]. [^3]: This might make the `ts` property within receipts redundant. We are not actually sure what purpose this property is intended to serve. -The server should include an `m.thread_id` property in any event that has an +The server should include a `thread_id` property in any event that has an ancestor relationship that includes an `m.thread` relationship. The value of -`m.thread_id` is the event ID of the event referenced by the `m.thread` +`thread_id` is the event ID of the event referenced by the `m.thread` relationship. This makes it explicit how the server has categorised events, meaning clients @@ -254,9 +296,9 @@ We propose that the definition of whether an event is read should be: > pointing at an event that is *in the same thread* as the event, and is *after > or the same as* the event. > -> Because the receipt itself contains the `m.order` of the pointed-to event, +> Because the receipt itself contains the `order` of the pointed-to event, > there is no need to examine the pointed-to event, so it is sufficient to -> compare the `m.order` of the event in question with the `m.order` in the +> compare the `order` of the event in question with the `order` in the > receipt. > > Redacted events are always read. @@ -267,10 +309,12 @@ Obviously, this definition depends on the definitions above. ## Potential issues -This special-cases threads over other relationships. +This special-cases threads over other relationships, raising them a little +closer to the same status as rooms. -But, this was already the case with threaded receipts, and we believe it is -necessary to provide consistent behaviour. +But, this was already the case with threaded receipts, and because of +`thread_id`'s special place in receipts, we believe it needs similar special +treatment in events to provide consistent behaviour. The change to consider thread roots (and reactions to them) as outside of the thread may be inconvenient for clients, because they will probably want to From c452342ff783151c40db4ac82fbd1dec511c8e34 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 13:44:30 +0100 Subject: [PATCH 15/51] Remove TODO list --- proposals/4033-event-thread-and-order.md | 18 ------------------ 1 file changed, 18 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 537eac612d1..c2c55e7af4f 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -1,23 +1,5 @@ # MSC4033: Providing thread and order for all events to allow consistent read receipt handling -## TODO - -- [x] review request Patrick -- [x] Mention no need for the words Stream Order to be included -- [x] Mention server-side calculation of unreadness -- [x] Acknowledge patrick and others -- [x] Make it robust to redactions by always considering redacted messages read -- [x] Remove thread roots from the thread -- [x] ~~Move ordering into unsigned (maybe thread id too?)~~ No - unsigned is frowned upon -- [x] Events can never have the same stream order -- [x] Reword to some generic order instead of stream order? -- [x] Can't do m.is_thread_root because we don't know until a child exists -- [x] Example of inconsistent Sync Order -- [x] Explicitly say we should update the spec wording around what we mean by read-up-to -- [x] Consider the thread root not being in the thread. Would need to think about whether it matters if the thread root is somehow later than a thread message in Stream Order. -- [x] JSON examples -- [x] Suggest that `ts` in receipts might be redundant? - We argue that we have made it unnecessarily hard for clients and servers to decide whether a message is read or unread, and we can solve this problem by clarifying the definitions, and including extra pieces of information with From 2d6a03efdf4949dcada1823c3bb52d994e06df5b Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 13:46:56 +0100 Subject: [PATCH 16/51] Wrap code blocks in pre --- proposals/4033-event-thread-and-order.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index c2c55e7af4f..4d73bcc8bbb 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -36,8 +36,8 @@ We do not propose to change the above definition. To perform this calculation we need definitions of *after or the same as* and *in the same thread*. -The meaning *the same as* is clear: if a receipt points at an event, then that -event is read. +The meaning of *the same as* is clear: if a receipt points at an event, then +that event is read. ### Current definition of *after* @@ -201,7 +201,7 @@ property when redacting events. Example event (changes are highlighted in bold): -{ +
{
   "content": {
     "body": "This is an example text message",
     "format": "org.matrix.custom.html",
@@ -218,11 +218,11 @@ Example event (changes are highlighted in bold):
     "age": 1234,
     "order": 56764334543
   }
-}
+}
 Example receipt (changes are highlighted in bold): -{ +
{
   "content": {
     "$1435641916114394fHBLK:matrix.org": {
       "order": 56764334544,
@@ -239,7 +239,7 @@ Example receipt (changes are highlighted in bold):
     }
   },
   "type": "m.receipt"
-}
+}
 We propose: From fba3bbed580f2256fda767d27380cf0d7cb21fa8 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 13:48:25 +0100 Subject: [PATCH 17/51] Format code with pre only --- proposals/4033-event-thread-and-order.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 4d73bcc8bbb..791c20f0dd1 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -201,7 +201,7 @@ property when redacting events. Example event (changes are highlighted in bold): -
{
+
{
   "content": {
     "body": "This is an example text message",
     "format": "org.matrix.custom.html",
@@ -218,11 +218,11 @@ Example event (changes are highlighted in bold):
     "age": 1234,
     "order": 56764334543
   }
-}

+}
Example receipt (changes are highlighted in bold): -
{
+
{
   "content": {
     "$1435641916114394fHBLK:matrix.org": {
       "order": 56764334544,
@@ -239,7 +239,7 @@ Example receipt (changes are highlighted in bold):
     }
   },
   "type": "m.receipt"
-}

+}
We propose: From f4899a039c0c911c0771bd8502869be496da3b64 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 13:49:48 +0100 Subject: [PATCH 18/51] Fix misplaced space --- proposals/4033-event-thread-and-order.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 791c20f0dd1..2d1970f95df 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -58,7 +58,7 @@ sync or received in a later sync, is after the other one [^1]. possible for different clients to receive events in a different order from each other for the same account. For example, if one client is doing an incremental sync, and another is doing an initial sync, recently-arrived - events that are "old "in Topological Order may be received in different orders + events that are "old" in Topological Order may be received in different orders on the two clients. See also [Spec Issue #1167](https://github.com/matrix-org/matrix-spec/issues/1167). From afa9629807c211a68adf1d345fa2ae30a1ee4418 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 13:50:16 +0100 Subject: [PATCH 19/51] Add missing "in" --- proposals/4033-event-thread-and-order.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 2d1970f95df..4ea45ef2e58 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -52,7 +52,7 @@ event for which it has a receipt via sync, then the event that was later in the sync or received in a later sync, is after the other one [^1]. [^1]: We think this is similar to Stream Ordering, which is mentioned once in - the spec without definition + the spec without definition in [7.6 Syncing](https://spec.matrix.org/unstable/client-server-api/#syncing), but we are not certain that it is identical, because we believe it may be possible for different clients to receive events in a different order from From 90e579865e67f2dd76d8a7764bc1e8e8b3a4f7ae Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 13:51:51 +0100 Subject: [PATCH 20/51] Explain what the spec issue is about --- proposals/4033-event-thread-and-order.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 4ea45ef2e58..e9c4a9a56e9 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -61,7 +61,8 @@ sync or received in a later sync, is after the other one [^1]. events that are "old" in Topological Order may be received in different orders on the two clients. -See also [Spec Issue #1167](https://github.com/matrix-org/matrix-spec/issues/1167). +See also [Spec Issue #1167](https://github.com/matrix-org/matrix-spec/issues/1167), +which calls out this ambiguity about the meaning of "read up to". ### Current definition of *in the same thread* From 76217a01c48adbeb2b3b780abd27284196a19034 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 13:53:33 +0100 Subject: [PATCH 21/51] Remove redundant part of main thread definition --- proposals/4033-event-thread-and-order.md | 1 - 1 file changed, 1 deletion(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index e9c4a9a56e9..58a261afa2b 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -79,7 +79,6 @@ An event is in thread A if: An event is in the main thread if: -* it is not in another thread, or * its ancestor events do not include an `m.thread` relationship Note that thread root events, and their non-threaded children (e.g. reactions to From c9bdac82f3c154cc2fbfb288382a5d91156c796e Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 13:57:37 +0100 Subject: [PATCH 22/51] Reflect the spec saying thread roots are not in main --- proposals/4033-event-thread-and-order.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 58a261afa2b..689ea25b087 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -77,12 +77,10 @@ An event is in thread A if: in the thread), or * it has an ancestor event (found by traversing relationships) that fits either of the above. -An event is in the main thread if: - -* its ancestor events do not include an `m.thread` relationship - -Note that thread root events, and their non-threaded children (e.g. reactions to -thread root events) are in BOTH the main thread and another thread. +The spec states that room events are in `main` if they are not in another thread, but +clients such as Element Web treat thread roots, and non-thread descendants of thread roots +(such as reactions to the thread root) as being in BOTH `main` and the thread branching +from the thread root. ### Problems with the current definitions From 501bfacd2034e607df6f8642e42d02d31845ed31 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 14:02:21 +0100 Subject: [PATCH 23/51] Restrict to just redacted events being always read --- proposals/4033-event-thread-and-order.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 689ea25b087..96ff4493b17 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -184,8 +184,7 @@ the current definition. ### Proposed change in consideration of redacted events -We propose that redacted events and redaction events should never be considered -unread. +We propose that redacted events should never be considered unread. This avoids the need to identify which thread a redacted event belongs to, which will be difficult if its `thread_id` property has been stripped out. From 17bfe74f910e13f69e6bbfde3539b0181e595759 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 14:05:02 +0100 Subject: [PATCH 24/51] Add missing "a" --- proposals/4033-event-thread-and-order.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 96ff4493b17..0ef4a2cdd74 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -256,7 +256,7 @@ relationship. This makes it explicit how the server has categorised events, meaning clients and servers can always agree on which thread an event is in, so the meaning of -threaded receipt is clear. +a threaded receipt is clear. It also makes it explicit which event is before or after another event in Stream Order. From c02285b5d2f7f0543166f0fe46f72eafc9a73569 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 14:07:41 +0100 Subject: [PATCH 25/51] Servers probably also think thread roots are in main --- proposals/4033-event-thread-and-order.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index 0ef4a2cdd74..f1a72499560 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -80,7 +80,9 @@ An event is in thread A if: The spec states that room events are in `main` if they are not in another thread, but clients such as Element Web treat thread roots, and non-thread descendants of thread roots (such as reactions to the thread root) as being in BOTH `main` and the thread branching -from the thread root. +from the thread root. We strongly suspect that home servers also consider thread roots +to be in the main thread, since otherwise their status would change when a new thread +reply was added. ### Problems with the current definitions From a48c1d0f565703531dd7d5ce1388cc4a414f48d2 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 15:07:02 +0100 Subject: [PATCH 26/51] Move thread_id into content thanks to conversation with Nico --- proposals/4033-event-thread-and-order.md | 21 +++++++++++---------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index f1a72499560..af646c2d22c 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -173,7 +173,7 @@ We propose that the definition of *in the same thread* should use this wording: An event is in a non-main-thread if: -* it has a `thread_id` property +* it has a `thread_id` property in its content Otherwise, it is in the `main` thread. @@ -206,11 +206,11 @@ Example event (changes are highlighted in bold): "format": "org.matrix.custom.html", "formatted_body": "<b>This is an example text message</b>", "msgtype": "m.text" + "thread_id": "$fgsUZKDJdhlrceRyhj", // Optional }, "event_id": "$143273582443PhrSn:example.org", "origin_server_ts": 1432735824653, "room_id": "!jEsUZKDJdhlrceRyVU:example.org", - "thread_id": "$fgsUZKDJdhlrceRyhj", // Optional "sender": "@example:example.org", "type": "m.room.message", "unsigned": { @@ -242,7 +242,7 @@ Example receipt (changes are highlighted in bold): We propose: -* all events in a thread should contain a `thread_id` property. +* all events in a thread should contain a `thread_id` property in their content. * all events should contain an `order` property. * all receipts should contain an `order` property alongside `m.read` and/or `m.read.private` inside the information about an event, which is a @@ -251,14 +251,14 @@ We propose: [^3]: This might make the `ts` property within receipts redundant. We are not actually sure what purpose this property is intended to serve. -The server should include a `thread_id` property in any event that has an -ancestor relationship that includes an `m.thread` relationship. The value of -`thread_id` is the event ID of the event referenced by the `m.thread` -relationship. +The creator of an event should include a `thread_id` property in the content of +any event that has an ancestor relationship that includes an `m.thread` +relationship. The value of `thread_id` is the event ID of the event referenced +by the `m.thread` relationship. -This makes it explicit how the server has categorised events, meaning clients -and servers can always agree on which thread an event is in, so the meaning of -a threaded receipt is clear. +This makes it explicit how events are categorised, meaning clients and servers +can always agree on which thread an event is in, so the meaning of a threaded +receipt is clear. It also makes it explicit which event is before or after another event in Stream Order. @@ -373,3 +373,4 @@ ideas from @t3chguy, @justjanne, @germain-gg and @weeman1337. * 2023-07-05 Give a definition of Stream Order * 2023-07-05 Be explicit about Stream Order not going over federation * 2023-07-05 Mention disagreeing about what another user has read +* 2023-07-05 Move thread_id into content after talking to @deepbluev7 From aeb06502e94470532c9918adf572db00ebacf2c7 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Wed, 5 Jul 2023 15:17:54 +0100 Subject: [PATCH 27/51] Mention 3051 as an alternative to thread_id --- proposals/4033-event-thread-and-order.md | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index af646c2d22c..eea0cc1883e 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -307,6 +307,20 @@ which the client can use to draw attention to the thread root. ## Alternatives +### Use MSC3501 instead of thread_id + +If [MSC3051: A scalable relation format](https://github.com/matrix-org/matrix-spec-proposals/pull/3051) +is on the path to standardisation, it could be used to specify the thread +containing each event. + +For the purposes of making receipts work, this is just as good as using +`thread_id`, and the author of this MSC supports MSC3051. + +We would simply need to mandate that anyone creating an event within a thread +must include an `m.thread` relation to that thread, even if the event is +a child of an event with a similar relation. This would be directly equivalent +to adding `thread_id` to the content. + ### This replaces other attempts to fix receipts This proposal would replace From d58758fa341912e9e8ed87911debe66d4a9adb73 Mon Sep 17 00:00:00 2001 From: Andy Balaam Date: Thu, 6 Jul 2023 17:04:45 +0100 Subject: [PATCH 28/51] Move thread_id to cleartext of content --- proposals/4033-event-thread-and-order.md | 25 +++++++++++++++++++----- 1 file changed, 20 insertions(+), 5 deletions(-) diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md index eea0cc1883e..6517329cd59 100644 --- a/proposals/4033-event-thread-and-order.md +++ b/proposals/4033-event-thread-and-order.md @@ -173,7 +173,7 @@ We propose that the definition of *in the same thread* should use this wording: An event is in a non-main-thread if: -* it has a `thread_id` property in its content +* it has a `thread_id` property in its cleartext content Otherwise, it is in the `main` thread. @@ -219,6 +219,21 @@ Example event (changes are highlighted in bold): } } +Example encrypted event (changes are highlighted in bold): + +
{
+    "type": "m.room.encrypted",
+    "content": {
+        "thread_id": "$fgsUZKDJdhlrceRyhj", // Optional
+        "algorithm": "m.megolm.v1.aes-sha2",
+        "sender_key": "",
+        "device_id": "",
+        "session_id": "",
+        "ciphertext": ""
+    }
+    // irrelevant fields not shown
+}
+ Example receipt (changes are highlighted in bold): 
{
@@ -251,10 +266,10 @@ We propose:
 [^3]: This might make the `ts` property within receipts redundant. We are not
   actually sure what purpose this property is intended to serve.
 
-The creator of an event should include a `thread_id` property in the content of
-any event that has an ancestor relationship that includes an `m.thread`
-relationship. The value of `thread_id` is the event ID of the event referenced
-by the `m.thread` relationship.
+The creator of an event should include a `thread_id` property in the cleartext
+content of any event that has an ancestor relationship that includes an
+`m.thread` relationship. The value of `thread_id` is the event ID of the event
+referenced by the `m.thread` relationship.
 
 This makes it explicit how events are categorised, meaning clients and servers
 can always agree on which thread an event is in, so the meaning of a threaded

From bbf1c94ab8e3fab549ca25f4d7a7af3a6f369557 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 6 Jul 2023 17:56:10 +0100
Subject: [PATCH 29/51] Cut down to just ordering

---
 proposals/4033-event-thread-and-order.md | 374 ++++++++---------------
 1 file changed, 129 insertions(+), 245 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 6517329cd59..ce3e1127fe5 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -1,9 +1,11 @@
-# MSC4033: Providing thread and order for all events to allow consistent read receipt handling
+# MSC4033: Explicit ordering of events for receipts
 
-We argue that we have made it unnecessarily hard for clients and servers to
-decide whether a message is read or unread, and we can solve this problem by
-clarifying the definitions, and including extra pieces of information with
-every event that help to identify its thread(s) and ordering.
+The [spec](https://spec.matrix.org/unstable/client-server-api/#receipts) states
+that receipts are "read-up-to" without explaining what order the events are in,
+so it is difficult to decide whether an event is before or after a receipt.
+
+We propose adding an explicit order number to all events, so that it is clear
+which events are read.
 
 ## Motivation
 
@@ -16,15 +18,14 @@ unread.
 
 Both of these tasks require us to decide whether a message is read or unread.
 
-To make this decision we have receipts. There are two types of receipt: threaded
-and unthreaded.
+To make this decision we have receipts.
 
 To decide whether an event is read, we use the following rule:
 
 > An event is read if the room contains an unthreaded receipt pointing at an
-> event which is *after or the same as* the event, or a threaded receipt
-> pointing at an event that is *in the same thread* as the event, and is *after
-> or the same as* the event.
+> event which is *after* the event, or a threaded receipt pointing at an event
+> that is in the same thread as the event, and is *after* or the same as the
+> event.
 >
 > Otherwise, it is unread.
 
@@ -33,11 +34,7 @@ consider either private or public read receipts.)
 
 We do not propose to change the above definition.
 
-To perform this calculation we need definitions of *after or the same as* and *in the same
-thread*.
-
-The meaning of *the same as* is clear: if a receipt points at an event, then
-that event is read.
+To perform this calculation we need a clear definition of *after*.
 
 ### Current definition of *after*
 
@@ -49,170 +46,60 @@ Clients like Element Web make the assumption that *after* means "after in Sync
 Order", where "Sync Order" means "the order in which I (the client) received the
 events from the server via sync", so if a client received an event and another
 event for which it has a receipt via sync, then the event that was later in the
-sync or received in a later sync, is after the other one [^1].
-
-[^1]: We think this is similar to Stream Ordering, which is mentioned once in
-  the spec without definition in
-  [7.6 Syncing](https://spec.matrix.org/unstable/client-server-api/#syncing),
-  but we are not certain that it is identical, because we believe it may be
-  possible for different clients to receive events in a different order from
-  each other for the same account. For example, if one client is doing an
-  incremental sync, and another is doing an initial sync, recently-arrived
-  events that are "old" in Topological Order may be received in different orders
-  on the two clients.
+sync or received in a later sync, is after the other one.
+
+See
+[room-dag-concepts](https://github.com/matrix-org/synapse/blob/develop/docs/development/room-dag-concepts.md#depth-and-stream-ordering)
+for some Synapse-specific information on Stream Order. In Synapse, Sync Order is
+expected to be identical to its concept of Stream Order.
 
 See also [Spec Issue #1167](https://github.com/matrix-org/matrix-spec/issues/1167),
 which calls out this ambiguity about the meaning of "read up to".
 
-### Current definition of *in the same thread*
-
-See [11.6.22 Threaded read
-receipts](https://spec.matrix.org/latest/client-server-api/#threaded-read-receipts),
-but a summary in looser language is:
-
-An event is in thread A if:
-
-* its event ID is A and it has `m.thread` children (i.e. it is the thread root), or
-* it has an `m.thread` relationship to the event with ID A (i.e. it is a message
-  in the thread), or
-* it has an ancestor event (found by traversing relationships) that fits either of the above.
-
-The spec states that room events are in `main` if they are not in another thread, but
-clients such as Element Web treat thread roots, and non-thread descendants of thread roots
-(such as reactions to the thread root) as being in BOTH `main` and the thread branching
-from the thread root. We strongly suspect that home servers also consider thread roots
-to be in the main thread, since otherwise their status would change when a new thread
-reply was added.
-
-### Problems with the current definitions
+### Problems with the current definition
 
 The current definition of *after* is ambiguous, and difficult for clients to
-calculate. It depends on only receiving events via sync, which is impractical
-due to backpagination, and the desire to fetch thread messages via the
+calculate. It depends on only receiving events via sync, which is impossible
+when we want to do backpagination or to fetch thread messages via the
 `relations` API.
 
-Further, we believe that the current working definition of "Sync Order" does not
-make sense, because different clients may receive events in different orders,
-meaning that a receipt from one client would be interpreted differently by
-another client.
-
-The current definition of *in the same thread* is difficult for clients to
-calculate because it requires fetching all parents of an event to identify which
-thread it belongs to. In the meantime, clients must deal with events and
-receipts that are "homeless".
-
-The current definitions also make it needlessly complex for clients to determine
-whether an event is read because they must fetch the event that is referred to by
-a receipt before they can make the decision, and hold the receipt "in limbo"
-before it is available.
+The current definition also makes it needlessly complex for clients to determine
+whether an event is read because the receipt itself does not hold enough
+information: the referenced event must be fetched and correctly ordered.
 
 ## Proposal
 
-We propose to modify the definitions slightly, and include extra information in
-the events provided by the server to make it easy to calculate whether an event
-is read.
-
-### Proposed definition of *after*
-
-We propose that the definition of *after* should be:
-
-* Event A is after event B if its *Stream Order* is larger.
-
-We define *Stream Order* to be an immutable, unique number attached to an event
-on creation that defines the order of events in regard to receipts.
-
-We propose updating the spec around receipts
-([11.6 Receipts](https://spec.matrix.org/latest/client-server-api/#receipts))
-to be explicit about what "read up to" means, using the above definition.
-
-#### Notes
-
-Some home servers already have a concept of Stream Order, and we intend
-that this definition is consistent with that as currently implemented. But, for
-the purposes of this proposal the only important aspect of Stream Order is that
-server and clients agree that receipts apply to events with a lower Stream
-Order, and that the Stream Order of an event never changes.
-
-If the name Stream Order proves confusing, this proposal can function equally
-well using a different name: it simply needs to be unique, immutable, and
-increase for "newer" messages.
-
-We do not require that Stream Order be consistent across federation (in fact, we
-believe that this would be impossible to achieve). The only requirement is that
-all the clients for a user and the one server to which they connect agree. For
-this reason, we propose that `order` be included in an event's `unsigned`
-property.
-
-### Notes on the meaning of Stream Order
-
-Because it controls the meaning of read receipts, it is desirable that Stream
-Order be as close as possible to Sync Order, the order in which clients receive
-events via sync. However, since clients receive events in different orders
-depending on the APIs they use, it is not a goal that Stream Order exactly match
-the order in which clients receive events. Instead, it provides a canonical
-order that means we can be clear about what the user has read, and thus should
-generally increase for "newer" messages. Clients may decide to re-order events
-into Stream Order, or they may decide to display unread messages higher up the
-timeline if the orders do not match the order they choose for display.
-
-Because Stream Order may be inconsistent across federation[^2], one user may
-occasionally see a different unread status for another user from what that user
-themselves see. We regard this as impossible to avoid, and expect that in most
-cases it will be unnoticeable, since home servers with good connectivity will
-normally have similar Stream Order. When servers have long network splits, there
-will be a noticeable difference at first, but once messages start flowing
-normally and users start reading them, the differences will disappear as new
-events will have higher Stream order than the older ones on both servers.
-
-[^2]: In theory, Stream Order could also be inconsistent across different users
-  on the same home server, although we expect in practice this will not happen.
-
-### Proposed definition of *in the same thread*
-
-We propose that the definition of *in the same thread* should use this wording:
-
-An event is in a non-main-thread if:
-
-* it has a `thread_id` property in its cleartext content
-
-Otherwise, it is in the `main` thread.
+We propose to add an explicit order number to events and receipts, so we always
+know what is before or after what.
 
-No events are in more than one thread.
+This order should be a number that is attached to an event by the server before
+it sends it to any client, and it should never change. It should,
+loosely-speaking, increase for "newer" messages. It may be negative.
 
-Note: this means that thread roots are in the `main` thread, and not in the
-thread branching from them. Non-thread children of thread roots (e.g.
-reactions to a thread root) are also in the `main` thread. This is a change to
-the current definition.
+The exact meaning of this order is not as important as the fact that it exists
+and is consistent between clients and servers, and events and receipts.
 
-### Proposed change in consideration of redacted events
+We do not require that order be consistent across federation. The only
+requirement is that all the clients for a user and the one server to which they
+connect agree. For this reason, we propose that `order` be included in an
+event's `unsigned` property.
 
-We propose that redacted events should never be considered unread.
-
-This avoids the need to identify which thread a redacted event belongs to, which
-will be difficult if its `thread_id` property has been stripped out.
-
-Since we propose that receipts contain the `order` of their referred-to event,
-this means we do not need to look within a redacted event for its Stream Order,
-because the receipt provides it. This avoids needing to preserve the `order`
-property when redacting events.
-
-### Supporting changes to event structure
+### Examples
 
 Example event (changes are highlighted in bold):
 
 
{
+  "type": "m.room.message",
   "content": {
     "body": "This is an example text message",
     "format": "org.matrix.custom.html",
     "formatted_body": "<b>This is an example text message</b>",
     "msgtype": "m.text"
-    "thread_id": "$fgsUZKDJdhlrceRyhj", // Optional
   },
   "event_id": "$143273582443PhrSn:example.org",
   "origin_server_ts": 1432735824653,
   "room_id": "!jEsUZKDJdhlrceRyVU:example.org",
   "sender": "@example:example.org",
-  "type": "m.room.message",
   "unsigned": {
     "age": 1234,
     "order": 56764334543
@@ -222,16 +109,22 @@ Example event (changes are highlighted in bold):
 Example encrypted event (changes are highlighted in bold):
 
 
{
-    "type": "m.room.encrypted",
-    "content": {
-        "thread_id": "$fgsUZKDJdhlrceRyhj", // Optional
-        "algorithm": "m.megolm.v1.aes-sha2",
-        "sender_key": "",
-        "device_id": "",
-        "session_id": "",
-        "ciphertext": ""
-    }
-    // irrelevant fields not shown
+  "type": "m.room.encrypted",
+  "content": {
+    "algorithm": "m.megolm.v1.aes-sha2",
+    "sender_key": "",
+    "device_id": "",
+    "session_id": "",
+    "ciphertext": ""
+  }
+  "event_id": "$143273582443PhrSn:example.org",
+  "origin_server_ts": 1432735824653,
+  "room_id": "!jEsUZKDJdhlrceRyVU:example.org",
+  "sender": "@example:example.org",
+  "unsigned": {
+    "age": 1234,
+    "order": 56764334543
+  }
 }

 
 Example receipt (changes are highlighted in bold):
@@ -240,16 +133,8 @@ Example receipt (changes are highlighted in bold):
   "content": {
     "$1435641916114394fHBLK:matrix.org": {
       "order": 56764334544,
-      "m.read": {
-        "@rikj:jki.re": {
-          "ts": 1436451550453
-        }
-      },
-      "m.read.private": {
-        "@self:example.org": {
-          "ts": 1661384801651
-        }
-      }
+      "m.read": { "@rikj:jki.re": { "ts": 1436451550453 } },
+      "m.read.private": { "@self:example.org": { "ts": 1661384801651 } }
     }
   },
   "type": "m.receipt"
@@ -257,92 +142,100 @@ Example receipt (changes are highlighted in bold):
 
 We propose:
 
-* all events in a thread should contain a `thread_id` property in their content.
-* all events should contain an `order` property.
-* all receipts should contain an `order` property alongside `m.read`
-  and/or `m.read.private` inside the information about an event, which is a
-  cache of the `order` property within the referred-to event [^3].
-
-[^3]: This might make the `ts` property within receipts redundant. We are not
-  actually sure what purpose this property is intended to serve.
+* all events should contain an `order` property inside `unsigned`.
+* all receipts should contain an `order` property alongside `m.read` inside the
+  information about an event, which is a cache of the `order` property within
+  the referred-to event.
 
-The creator of an event should include a `thread_id` property in the cleartext
-content of any event that has an ancestor relationship that includes an
-`m.thread` relationship. The value of `thread_id` is the event ID of the event
-referenced by the `m.thread` relationship.
+### Proposed definition of *after*
 
-This makes it explicit how events are categorised, meaning clients and servers
-can always agree on which thread an event is in, so the meaning of a threaded
-receipt is clear.
+We propose that the definition of *after* should be:
 
-It also makes it explicit which event is before or after another event in Stream
-Order.
+* Event A is after event B if its order is larger.
 
-This prevents disagreements between clients and servers about the meaning of a
-particular read receipt, and by including the Stream Order of the referred event
-in the receipt, avoids the need to fetch that event in order to make a decision
-about which events are read.
+We propose updating the spec around receipts
+([11.6 Receipts](https://spec.matrix.org/latest/client-server-api/#receipts))
+to be explicit about what "read up to" means, using the above definition.
 
 ### Definition of read and unread events
 
-We propose that the definition of whether an event is read should be:
+We propose that the definition of whether an event is read should be unchanged,
+but this note should be added:
 
-> An event is read if the room contains an unthreaded receipt pointing at an
-> event which is *after or the same as* the event, or a threaded receipt
-> pointing at an event that is *in the same thread* as the event, and is *after
-> or the same as* the event.
->
-> Because the receipt itself contains the `order` of the pointed-to event,
+> (Because the receipt itself contains the `order` of the pointed-to event,
 > there is no need to examine the pointed-to event, so it is sufficient to
 > compare the `order` of the event in question with the `order` in the
-> receipt.
->
-> Redacted events are always read.
->
-> Otherwise, it is unread.
+> receipt.)
 
-Obviously, this definition depends on the definitions above.
+And the above definition of *after* should be included.
 
-## Potential issues
+### Redacted events
 
-This special-cases threads over other relationships, raising them a little
-closer to the same status as rooms.
+Since we propose that receipts should cache the `order` of their referred-to
+event, this means we do not need to look within a redacted event for its order,
+because the receipt provides it. This avoids needing to preserve the `order`
+property when redacting events.
 
-But, this was already the case with threaded receipts, and because of
-`thread_id`'s special place in receipts, we believe it needs similar special
-treatment in events to provide consistent behaviour.
+## Discussion
 
-The change to consider thread roots (and reactions to them) as outside of the
-thread may be inconvenient for clients, because they will probably want to
-display those events in any "thread view" they display. We consider this
-inconvenience worthwhile because it is necessary to ensure the semantics of read
-receipts make sense. We do not think that reactions or edits to a thread root
-should mark that thread as unread - instead they mark the main thread as unread,
-which the client can use to draw attention to the thread root.
+### What order to display events in the UI?
 
-## Alternatives
+It is desirable that the order property should match the order of events
+displayed in the client as closely as possible, so that receipts behave
+consistently with the displayed timeline. However, clients may have different
+ideas about where to display late-arriving messages, so it is impossible to
+define an order that works for all clients. Instead we agree that a consistent
+answer is the best we can do, and rely on clients to provide the best UX they
+can for late-arriving messages.
+
+### Stream order or Topological Order?
+
+The two orders that we might choose to populate the `order` property are "stream
+order" where late-arriving messages tend to receive higher numbers, or
+"Topological Order" where late-arriving message tend to receive lower numbers.
+
+We believe that it is better to consider late-arriving messages as unread,
+meaning the client has the information that these newly arrived messages have
+not been read and can choose how to display it (or not). This is what leads us
+to suggest Stream Order as the correct choice.
+
+However, if servers choose Topological Order, this proposal still works - we
+just have what the authors consider undesirable behaviour regarding
+late-arriving events (they are seen as read even though they are not).
+
+### Inconsistency across federation
+
+Because order may be inconsistent across federation[^1], one user may
+occasionally see a different unread status for another user from what that user
+themselves see. We regard this as impossible to avoid, and expect that in most
+cases it will be unnoticeable, since home servers with good connectivity will
+normally see events in similar orders. When servers have long network splits,
+there will be a noticeable difference at first, but once messages start flowing
+normally and users start reading them, the differences will disappear as new
+events will have higher Stream order than the older ones on both servers.
+
+[^1]: In fact, order could also be inconsistent across different users on the
+  same home server, although we expect in practice this will not happen.
+
+The focus of this proposal is that a single user sees consistent behaviour
+around their own read receipts, and we consider that much more important that
+the edge case of inconsistent behaviour across federation after a network split.
 
-### Use MSC3501 instead of thread_id
+## Implementation Notes
 
-If [MSC3051: A scalable relation format](https://github.com/matrix-org/matrix-spec-proposals/pull/3051)
-is on the path to standardisation, it could be used to specify the thread
-containing each event.
+Some home servers such as Synapse already have a concept of Stream Order. We
+expect that the order defined here could be implemented using Stream Order.
 
-For the purposes of making receipts work, this is just as good as using
-`thread_id`, and the author of this MSC supports MSC3051.
+## Potential issues
 
-We would simply need to mandate that anyone creating an event within a thread
-must include an `m.thread` relation to that thread, even if the event is
-a child of an event with a similar relation. This would be directly equivalent
-to adding `thread_id` to the content.
+This explicitly allows receipts to be inconsistent across federation. In
+practice this is already the case in the wild, and is impossible to solve using
+Stream Order. The problems with using Topological Order (and Sync Order) have
+already been outlined.
 
-### This replaces other attempts to fix receipts
+## Alternatives
 
-This proposal would replace
-[MSC4023: Thread ID for 2nd order-relation](https://github.com/matrix-org/matrix-spec-proposals/pull/4023)
-and the idea that
-[MSC3051: A scalable relation format](https://github.com/matrix-org/matrix-spec-proposals/pull/3051)
-would help fix receipts.
+### Solves the same problem MSC3981 Relations Recursion tried to solve
 
 This propsal would not replace
 [MSC3981: /relations recursion](https://github.com/matrix-org/matrix-spec-proposals/pull/3981)
@@ -356,17 +249,7 @@ would be returned in Sync Order, but this is not true: the proposal will return
 events in Topological Order, which is useless for determining which events are
 read.
 
-### Avoiding saying "Stream Order"
-
-We could avoid the phrase Stream Order in this proposal, and instead simply talk
-about a consistent order that the server and client agree on because it is
-included with event info.
-
-We could even note that we expect it to be Stream Order for homeservers that
-have such a concept, but the important thing for us is that it is consistent and
-explicit.
-
-### The server calculates unread status
+### The server could calculate unread status
 
 We could use the definitions within this proposal but avoid calculating what was
 unread on the client. Instead we could ask the server to figure out which rooms
@@ -403,3 +286,4 @@ ideas from @t3chguy, @justjanne, @germain-gg and @weeman1337.
 * 2023-07-05 Be explicit about Stream Order not going over federation
 * 2023-07-05 Mention disagreeing about what another user has read
 * 2023-07-05 Move thread_id into content after talking to @deepbluev7
+* 2023-07-06 Reduced to just order. Thread IDs will be a separate MSC

From 36a28e5c2b0bb43bf02627a3fc7022d16e0680f4 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Fri, 7 Jul 2023 10:56:31 +0100
Subject: [PATCH 30/51] Small brevity improvements

---
 proposals/4033-event-thread-and-order.md | 15 +++++----------
 1 file changed, 5 insertions(+), 10 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index ce3e1127fe5..63a98bbf548 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -9,18 +9,15 @@ which events are read.
 
 ## Motivation
 
-In order to decide whether a room is unread, a Matrix client must decide whether
-it contains any unread messages.
+To decide whether a room is unread, a Matrix client must decide whether it
+contains any unread messages.
 
-Similarly, in order to decide whether a room has notifications, a Matrix client
-or server must decide whether any of its potentially-notifying messages is
-unread.
+Similarly, to decide whether a room has notifications, we must decide whether
+any of its potentially-notifying messages is unread.
 
 Both of these tasks require us to decide whether a message is read or unread.
 
-To make this decision we have receipts.
-
-To decide whether an event is read, we use the following rule:
+To make this decision we have receipts. We use the following rule:
 
 > An event is read if the room contains an unthreaded receipt pointing at an
 > event which is *after* the event, or a threaded receipt pointing at an event
@@ -32,8 +29,6 @@ To decide whether an event is read, we use the following rule:
 (In both cases we only consider receipts sent by the current user, obviously. We
 consider either private or public read receipts.)
 
-We do not propose to change the above definition.
-
 To perform this calculation we need a clear definition of *after*.
 
 ### Current definition of *after*

From be77a585389af5f499092af63d080498f98853b3 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Fri, 7 Jul 2023 11:01:52 +0100
Subject: [PATCH 31/51] Note that order should be inserted by servers

---
 proposals/4033-event-thread-and-order.md | 31 ++++++++++++++++++++++--
 1 file changed, 29 insertions(+), 2 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 63a98bbf548..cf9250e182b 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -127,8 +127,8 @@ Example receipt (changes are highlighted in bold):
 
{
   "content": {
     "$1435641916114394fHBLK:matrix.org": {
-      "order": 56764334544,
-      "m.read": { "@rikj:jki.re": { "ts": 1436451550453 } },
+      "order": 56764334544, // Optional
+      "m.read": { "@rikj:jki.re": { "ts": 1436451550453, "thread_id": "$x" } },
       "m.read.private": { "@self:example.org": { "ts": 1661384801651 } }
     }
   },
@@ -142,6 +142,11 @@ We propose:
   information about an event, which is a cache of the `order` property within
   the referred-to event.
 
+The `order` property in receipts should be inserted by servers when they are
+creating the aggregated receipt event. If the server does not have the
+referenced event and so does not know its order, this property may be omitted,
+and clients will need to look up the event themselves to determine its order.
+
 ### Proposed definition of *after*
 
 We propose that the definition of *after* should be:
@@ -255,6 +260,28 @@ notifications that are encrypted when they pass through the server, so this
 proposal would probably be unaltered even if we added the capability for servers
 to surface which rooms are unread.
 
+### Location of order property in receipts
+
+It is possible that adding an `order` property within receipts alongside
+`m.read` might break existing clients. If so, an alternative would be to include
+it alongside `ts`, which is probably less likely to cause problems.
+
+Example:
+
+
{
+  "content": {
+    "$1435641916114394fHBLK:matrix.org": {
+      "m.read": {
+        "@rikj:jki.re": {
+          "ts": 1436451550453,
+          "order": 56764334544, // Optional
+        }
+      },
+    }
+  },
+  "type": "m.receipt"
+}

+
 ## Security considerations
 
 None highlighted so far.

From 04c4606a1b8173aa830284e3c303a4323ab0a167 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Fri, 7 Jul 2023 15:52:54 +0100
Subject: [PATCH 32/51] Move order to be a sibling of ts in receipts

---
 proposals/4033-event-thread-and-order.md | 37 ++++++++++++++----------
 1 file changed, 22 insertions(+), 15 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index cf9250e182b..212dabd6055 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -127,9 +127,12 @@ Example receipt (changes are highlighted in bold):
 
{
   "content": {
     "$1435641916114394fHBLK:matrix.org": {
-      "order": 56764334544, // Optional
-      "m.read": { "@rikj:jki.re": { "ts": 1436451550453, "thread_id": "$x" } },
-      "m.read.private": { "@self:example.org": { "ts": 1661384801651 } }
+      "m.read": {
+        "@rikj:jki.re": {
+          "ts": 1436451550453,
+          "order": 56764334544, // Optional
+        }
+      },
     }
   },
   "type": "m.receipt"
@@ -138,7 +141,7 @@ Example receipt (changes are highlighted in bold):
 We propose:
 
 * all events should contain an `order` property inside `unsigned`.
-* all receipts should contain an `order` property alongside `m.read` inside the
+* all receipts should contain an `order` property alongside `ts` inside the
   information about an event, which is a cache of the `order` property within
   the referred-to event.
 
@@ -147,6 +150,11 @@ creating the aggregated receipt event. If the server does not have the
 referenced event and so does not know its order, this property may be omitted,
 and clients will need to look up the event themselves to determine its order.
 
+Note that the `order` property for a particular event will probably be the same
+for every user, so will be repeated multiple times in an aggregated receipt
+event. This structure was chosen to reduce the chance of breaking existing
+clients by introducing `order` at a higher level.
+
 ### Proposed definition of *after*
 
 We propose that the definition of *after* should be:
@@ -262,26 +270,24 @@ to surface which rooms are unread.
 
 ### Location of order property in receipts
 
-It is possible that adding an `order` property within receipts alongside
-`m.read` might break existing clients. If so, an alternative would be to include
-it alongside `ts`, which is probably less likely to cause problems.
-
-Example:
+Initially, we included `order` as a sibling of `m.read` inside the content of a
+receipt:
 
 
{
   "content": {
     "$1435641916114394fHBLK:matrix.org": {
-      "m.read": {
-        "@rikj:jki.re": {
-          "ts": 1436451550453,
-          "order": 56764334544, // Optional
-        }
-      },
+      "order": 56764334544, // Optional
+      "m.read": { "@rikj:jki.re": { "ts": 1436451550453, "thread_id": "$x" } },
+      "m.read.private": { "@self:example.org": { "ts": 1661384801651 } }
     }
   },
   "type": "m.receipt"
 }

 
+We moved it inside the content, as a sibling to `ts`, because multiple existing
+clients (mautrix-go, mautrix-python and matrix-rust-sdk) would have failed to
+parse the above JSON if they encountered it without first being updated.
+
 ## Security considerations
 
 None highlighted so far.
@@ -309,3 +315,4 @@ ideas from @t3chguy, @justjanne, @germain-gg and @weeman1337.
 * 2023-07-05 Mention disagreeing about what another user has read
 * 2023-07-05 Move thread_id into content after talking to @deepbluev7
 * 2023-07-06 Reduced to just order. Thread IDs will be a separate MSC
+* 2023-07-06 Moved order deeper within receipts to reduce existing client impact

From 6e027e2a1161d848b6b6a131fabf93c1bf70e7d5 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 13 Jul 2023 14:19:24 +0100
Subject: [PATCH 33/51] Reword motivation paragraph

Co-authored-by: Patrick Cloke 
---
 proposals/4033-event-thread-and-order.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 212dabd6055..25c69afaaa6 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -64,8 +64,8 @@ information: the referenced event must be fetched and correctly ordered.
 
 ## Proposal
 
-We propose to add an explicit order number to events and receipts, so we always
-know what is before or after what.
+We propose to add an explicit order number to events and receipts, so we can easily
+compare whether an event is before or after a receipt.
 
 This order should be a number that is attached to an event by the server before
 it sends it to any client, and it should never change. It should,

From 5499b04ef35285f748c626a7935948bb2c6865b8 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 13 Jul 2023 14:23:14 +0100
Subject: [PATCH 34/51] Formatting

---
 proposals/4033-event-thread-and-order.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 25c69afaaa6..3d6b1f9418a 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -64,8 +64,8 @@ information: the referenced event must be fetched and correctly ordered.
 
 ## Proposal
 
-We propose to add an explicit order number to events and receipts, so we can easily
-compare whether an event is before or after a receipt.
+We propose to add an explicit order number to events and receipts, so we can
+easily compare whether an event is before or after a receipt.
 
 This order should be a number that is attached to an event by the server before
 it sends it to any client, and it should never change. It should,

From 74933916096df2b21752202191d93913af38bbfb Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 13 Jul 2023 14:24:45 +0100
Subject: [PATCH 35/51] Mention the specific API used to get older messages

---
 proposals/4033-event-thread-and-order.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 3d6b1f9418a..6b9e019b628 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -55,8 +55,8 @@ which calls out this ambiguity about the meaning of "read up to".
 
 The current definition of *after* is ambiguous, and difficult for clients to
 calculate. It depends on only receiving events via sync, which is impossible
-when we want to do backpagination or to fetch thread messages via the
-`relations` API.
+since we sometimes want messages that did not arrive via sync, so we use
+different APIs such as `messages` or `relations`.
 
 The current definition also makes it needlessly complex for clients to determine
 whether an event is read because the receipt itself does not hold enough

From 7d3df448926440370635d442a2e5663e889a03e1 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 13 Jul 2023 14:27:23 +0100
Subject: [PATCH 36/51] Make clear the order only needs to increase relative to
 messages in the same room.

Co-authored-by: Patrick Cloke 
---
 proposals/4033-event-thread-and-order.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 6b9e019b628..33832f28d71 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -69,7 +69,7 @@ easily compare whether an event is before or after a receipt.
 
 This order should be a number that is attached to an event by the server before
 it sends it to any client, and it should never change. It should,
-loosely-speaking, increase for "newer" messages. It may be negative.
+loosely-speaking, increase for "newer" messages within the same room. It may be negative.
 
 The exact meaning of this order is not as important as the fact that it exists
 and is consistent between clients and servers, and events and receipts.

From c45a5e4f4855d071c8609d2716af52fb67b82fbc Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 13 Jul 2023 14:28:05 +0100
Subject: [PATCH 37/51] Formatting

---
 proposals/4033-event-thread-and-order.md | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 33832f28d71..9ae409559f7 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -69,7 +69,8 @@ easily compare whether an event is before or after a receipt.
 
 This order should be a number that is attached to an event by the server before
 it sends it to any client, and it should never change. It should,
-loosely-speaking, increase for "newer" messages within the same room. It may be negative.
+loosely-speaking, increase for "newer" messages within the same room. It may be
+negative.
 
 The exact meaning of this order is not as important as the fact that it exists
 and is consistent between clients and servers, and events and receipts.

From 1b668ce9ea33bd2c32906f4244eec47329f5a5aa Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 13 Jul 2023 14:38:47 +0100
Subject: [PATCH 38/51] Clarify the consistency guarantees and meaning of order

---
 proposals/4033-event-thread-and-order.md | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 9ae409559f7..57aa58e2b96 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -72,13 +72,15 @@ it sends it to any client, and it should never change. It should,
 loosely-speaking, increase for "newer" messages within the same room. It may be
 negative.
 
-The exact meaning of this order is not as important as the fact that it exists
-and is consistent between clients and servers, and events and receipts.
-
-We do not require that order be consistent across federation. The only
-requirement is that all the clients for a user and the one server to which they
-connect agree. For this reason, we propose that `order` be included in an
-event's `unsigned` property.
+The ordering must be consistent between a user's homeserver and all of that
+user's connected clients. There are no guarantees it is consistent across
+different users or rooms. It will be inconsistent across federation as there is
+no mechanism to sync order between homeservers. For this reason, we propose that
+`order` be included in an event's `unsigned` property.
+
+This proposal attaches no particular meaning to the rate at which the ordering
+increments. (Although we can imagine that some future proposal might want to
+expand this idea to include some meaning.)
 
 ### Examples
 

From d2fe0f4109fdb3b7b099b36e0ffc54421c485a72 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 13 Jul 2023 14:58:05 +0100
Subject: [PATCH 39/51] Include order in redacted events too

---
 proposals/4033-event-thread-and-order.md | 11 +++++++----
 1 file changed, 7 insertions(+), 4 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 57aa58e2b96..c1823b059cc 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -182,10 +182,12 @@ And the above definition of *after* should be included.
 
 ### Redacted events
 
-Since we propose that receipts should cache the `order` of their referred-to
-event, this means we do not need to look within a redacted event for its order,
-because the receipt provides it. This avoids needing to preserve the `order`
-property when redacting events.
+Existing servers already include an `unsigned` section with redacted events,
+despite `unsigned` not being mentioned in the [redaction
+rules](https://spec.matrix.org/unstable/rooms/v10/#redactions).
+
+Therefore we propose that redacted events should include `order` in exactly the
+same way as all room events.
 
 ## Discussion
 
@@ -319,3 +321,4 @@ ideas from @t3chguy, @justjanne, @germain-gg and @weeman1337.
 * 2023-07-05 Move thread_id into content after talking to @deepbluev7
 * 2023-07-06 Reduced to just order. Thread IDs will be a separate MSC
 * 2023-07-06 Moved order deeper within receipts to reduce existing client impact
+* 2023-07-13 Include order with redacted events after comments from @clokep

From 694317e27d826670e05a240284be027bc178a096 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 13 Jul 2023 16:23:55 +0100
Subject: [PATCH 40/51] Emphasise the room level when talking about server
 calculation

---
 proposals/4033-event-thread-and-order.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index c1823b059cc..6e2f85b424c 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -262,7 +262,7 @@ would be returned in Sync Order, but this is not true: the proposal will return
 events in Topological Order, which is useless for determining which events are
 read.
 
-### The server could calculate unread status
+### The server could report which rooms are unread
 
 We could use the definitions within this proposal but avoid calculating what was
 unread on the client. Instead we could ask the server to figure out which rooms

From be0f7ace1009d697059cac237ea4bfa39043f8f4 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Mon, 17 Jul 2023 13:26:11 +0100
Subject: [PATCH 41/51] Add a note that this applies to other users' receipts
 too

---
 proposals/4033-event-thread-and-order.md | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 6e2f85b424c..72d6cbb6c96 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -62,6 +62,11 @@ The current definition also makes it needlessly complex for clients to determine
 whether an event is read because the receipt itself does not hold enough
 information: the referenced event must be fetched and correctly ordered.
 
+Note: these problems actually apply to all receipts, not just those of the
+current user. The symptoms are much more visible and impactful when the current
+user's receipts are misinterpreted than for other users, but this proposal
+covers both cases.
+
 ## Proposal
 
 We propose to add an explicit order number to events and receipts, so we can

From e650a11aa5db1c53bd0a41a983377993e979fa9a Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Mon, 17 Jul 2023 13:29:11 +0100
Subject: [PATCH 42/51] Acknowledge that we actually are changing the
 definition of read/unread

---
 proposals/4033-event-thread-and-order.md | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 72d6cbb6c96..f80c8cd5ce5 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -175,15 +175,13 @@ to be explicit about what "read up to" means, using the above definition.
 
 ### Definition of read and unread events
 
-We propose that the definition of whether an event is read should be unchanged,
-but this note should be added:
+We propose that the definition of whether an event is read should include the
+original definition plus the above definition of *after*, and also include this
+clarification:
 
 > (Because the receipt itself contains the `order` of the pointed-to event,
-> there is no need to examine the pointed-to event, so it is sufficient to
-> compare the `order` of the event in question with the `order` in the
-> receipt.)
-
-And the above definition of *after* should be included.
+> there is no need to examine the pointed-to event: it is sufficient to compare
+> the `order` of the event in question with the `order` in the receipt.)
 
 ### Redacted events
 

From 7d1728e9d0462a6eb250b8bc9907ed7717b217cb Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 3 Aug 2023 15:57:45 +0100
Subject: [PATCH 43/51] Fix typo

Co-authored-by: Matthew Hodgson 
---
 proposals/4033-event-thread-and-order.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index f80c8cd5ce5..143c89a85b9 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -253,7 +253,7 @@ already been outlined.
 
 ### Solves the same problem MSC3981 Relations Recursion tried to solve
 
-This propsal would not replace
+This proposal would not replace
 [MSC3981: /relations recursion](https://github.com/matrix-org/matrix-spec-proposals/pull/3981)
 but would make it less important, because we would no longer depend on the
 server providing messages in Sync Order, so we could happily fetch messages

From 12cde91b2a3f3bb40af6ff480cf6253d4a3d079f Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 3 Aug 2023 16:02:48 +0100
Subject: [PATCH 44/51] Fix missing e in mxid

Co-authored-by: Matthew Hodgson 
---
 proposals/4033-event-thread-and-order.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 143c89a85b9..23fe078851c 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -136,7 +136,7 @@ Example receipt (changes are highlighted in bold):
   "content": {
     "$1435641916114394fHBLK:matrix.org": {
       "m.read": {
-        "@rikj:jki.re": {
+        "@erikj:jki.re": {
           "ts": 1436451550453,
           "order": 56764334544, // Optional
         }

From 6950a593f04ad00ea8e28b32c412480d3cfdd841 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Fri, 4 Aug 2023 15:41:18 +0100
Subject: [PATCH 45/51] Make order on a receipt mandatory

---
 proposals/4033-event-thread-and-order.md | 16 +++++++++++-----
 1 file changed, 11 insertions(+), 5 deletions(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 23fe078851c..c4008a0411f 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -138,7 +138,7 @@ Example receipt (changes are highlighted in bold):
       "m.read": {
         "@erikj:jki.re": {
           "ts": 1436451550453,
-          "order": 56764334544, // Optional
+          "order": 56764334544,
         }
       },
     }
@@ -154,9 +154,15 @@ We propose:
   the referred-to event.
 
 The `order` property in receipts should be inserted by servers when they are
-creating the aggregated receipt event. If the server does not have the
-referenced event and so does not know its order, this property may be omitted,
-and clients will need to look up the event themselves to determine its order.
+creating the aggregated receipt event.
+
+If the server is not able to provide the order of a receipt (e.g. because it
+does not have the relevant event) it should not send the receipt. If a server
+later receives an event, allowing it to provide an order for this receipt, it
+should send the receipt at that time. Rationale: without the order, a receipt is
+not useful to the client since it is not able to use it to determine which
+events are read. If a receipt points at an unknown event, the safest assumption
+is that other events in the room are unread i.e. there is no receipt.
 
 Note that the `order` property for a particular event will probably be the same
 for every user, so will be repeated multiple times in an aggregated receipt
@@ -284,7 +290,7 @@ receipt:
 
{
   "content": {
     "$1435641916114394fHBLK:matrix.org": {
-      "order": 56764334544, // Optional
+      "order": 56764334544,
       "m.read": { "@rikj:jki.re": { "ts": 1436451550453, "thread_id": "$x" } },
       "m.read.private": { "@self:example.org": { "ts": 1661384801651 } }
     }

From 2543428576f6bb657baec46e8b04e0adc523650c Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Fri, 4 Aug 2023 15:48:09 +0100
Subject: [PATCH 46/51] Document the idea of just never sending receipts that
 don't have order when we get them

---
 proposals/4033-event-thread-and-order.md | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index c4008a0411f..832319eab43 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -302,6 +302,19 @@ We moved it inside the content, as a sibling to `ts`, because multiple existing
 clients (mautrix-go, mautrix-python and matrix-rust-sdk) would have failed to
 parse the above JSON if they encountered it without first being updated.
 
+### Drop receipts with missing order information
+
+In the case where a server has a receipt to send to the client, but does not
+have the event to which it refers, and therefore cannot find its order, we
+proposed above that the server should hold the receipt until it has the relevant
+event, and send it then.
+
+Alternatively, we could simply never send the receipt under these circumstances.
+We believe that this is reasonable because it is not expected to happen for the
+user's own events, which are the most critical to provide accurate read
+receipts, and implementing the "hold and send later" strategy may cause extra
+work for the server for little practical gain.
+
 ## Security considerations
 
 None highlighted so far.

From 605eaddf12f8487d5d3c1e8bcad442816754ce8a Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 30 Nov 2023 17:35:19 +0000
Subject: [PATCH 47/51] Add a note about fully-read markers being out of scope

---
 proposals/4033-event-thread-and-order.md | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 832319eab43..a614764bc0c 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -7,6 +7,10 @@ so it is difficult to decide whether an event is before or after a receipt.
 We propose adding an explicit order number to all events, so that it is clear
 which events are read.
 
+This proposal covers receipts, and not fully-read markers. Fully-read markers
+have the same issue in terms of ordering, and should probably be fixed in a
+similar way, but they are not addressed here.
+
 ## Motivation
 
 To decide whether a room is unread, a Matrix client must decide whether it

From d749fb125a34d5dad864690abf5cc00aa89678c5 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 30 Nov 2023 17:44:30 +0000
Subject: [PATCH 48/51] Add notes about negative event order

---
 proposals/4033-event-thread-and-order.md | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index a614764bc0c..8154e10e16e 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -78,7 +78,10 @@ easily compare whether an event is before or after a receipt.
 
 This order should be a number that is attached to an event by the server before
 it sends it to any client, and it should never change. It should,
-loosely-speaking, increase for "newer" messages within the same room. It may be
+loosely-speaking, increase for "newer" messages within the same room.
+
+The order of an event may be negative, and if so it is understood that this
+event is always read. The order included with a receipt should never be
 negative.
 
 The ordering must be consistent between a user's homeserver and all of that
@@ -168,6 +171,10 @@ not useful to the client since it is not able to use it to determine which
 events are read. If a receipt points at an unknown event, the safest assumption
 is that other events in the room are unread i.e. there is no receipt.
 
+If a receipt is received for an event with negative order, the server should set
+the order in the receipt to zero. All events with negative order are understood
+to be read.
+
 Note that the `order` property for a particular event will probably be the same
 for every user, so will be repeated multiple times in an aggregated receipt
 event. This structure was chosen to reduce the chance of breaking existing

From 6859b8dbb0dc37f1a8a76249151aee15049dde17 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 30 Nov 2023 17:47:23 +0000
Subject: [PATCH 49/51] Add a note about negative order meaning event is read

---
 proposals/4033-event-thread-and-order.md | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 8154e10e16e..8fb1a1eb08a 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -200,6 +200,9 @@ clarification:
 > there is no need to examine the pointed-to event: it is sufficient to compare
 > the `order` of the event in question with the `order` in the receipt.)
 
+Further, it should be stated that events with negative order are always read,
+even if no receipt exists.
+
 ### Redacted events
 
 Existing servers already include an `unsigned` section with redacted events,

From d2cc49df9f4584e067db740969e899cc7871ba4b Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Thu, 30 Nov 2023 17:58:06 +0000
Subject: [PATCH 50/51] Add a note about order not being identical

---
 proposals/4033-event-thread-and-order.md | 26 ++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 8fb1a1eb08a..53f2138b15a 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -203,6 +203,32 @@ clarification:
 Further, it should be stated that events with negative order are always read,
 even if no receipt exists.
 
+### Order does not have to be unique
+
+If this proposal required the `order` property to be unique within a room, it
+might inadvertantly put constraints on the implementation of servers since some
+linearised process would need to be involved.
+
+So, we do not require that `order` should be unique within a room. Instead, if
+two events have the same `order`, they are both marked as read by a receipt with
+that order.
+
+Events with identical order introduce some imprecision into the process of
+marking events as read, so they should be minimised where possible, but some
+overlap is tolerable where the server implementation requires it.
+
+So, a server might choose to use the epoch millisecond at which it received a
+message as its order. However, if a server receives a large batch of messages in
+the same millisecond, this might cause undesirable behaviour, so a refinement
+might be the millisecond as the integer part and a fractional part that
+increases as the batch is processed, preserving the order in which the server
+receives the messages in the batch.
+
+If a server were processing multiple batches in parallel, it could implement
+this in each process separately, and accept that some events would receive
+identical orders, but this would be rare in practice and have little effect on
+end users' experience of unread markers.
+
 ### Redacted events
 
 Existing servers already include an `unsigned` section with redacted events,

From c7a8192d703e8f080ffe6b95bc6e97e297958281 Mon Sep 17 00:00:00 2001
From: Andy Balaam 
Date: Fri, 1 Dec 2023 11:02:52 +0000
Subject: [PATCH 51/51] Fix spelling error

---
 proposals/4033-event-thread-and-order.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/proposals/4033-event-thread-and-order.md b/proposals/4033-event-thread-and-order.md
index 53f2138b15a..3406042b6f3 100644
--- a/proposals/4033-event-thread-and-order.md
+++ b/proposals/4033-event-thread-and-order.md
@@ -206,7 +206,7 @@ even if no receipt exists.
 ### Order does not have to be unique
 
 If this proposal required the `order` property to be unique within a room, it
-might inadvertantly put constraints on the implementation of servers since some
+might inadvertently put constraints on the implementation of servers since some
 linearised process would need to be involved.
 
 So, we do not require that `order` should be unique within a room. Instead, if