Add HA and read & write expectations from event stores

[skleanthous]

Update expectations from event stores to require support for high availability scenarios, and at minimum require quorum writes, and causally consistent monotonic reads.

[ylorph]

Are those must have or really , really nice to have

[skleanthous]

It depends on perspective tbh. In production environment, such a setup can provide availability which is more that just nice to have; I am not aware of any server product that runs on a single instance of a database.

[skleanthous]

What I'm trying to say with the above is that this is a must-have guarantee from a database for a lot of services needing to store information. At the same time, it's not needed at all (it's not even a nice to have) for other (rarer ime) scenarios.

I'm happy to move that to bonuses if you prefer.

[ylorph]

And I'm not sure what you mean by causally consistent, monotonic reads

[skleanthous]

Let's assume event A happened and event A causes event B, and then event B causes event C. Then in a HA scenario:

• Causally consistent guarantees ensure that whichever server we hit, we would always see A before B or C, and we'd see B before C
• With monotonic reads guarantee, if on a first query we got [A, B], we are then guaranteed that we would never get just [A] on any subsequent query (potentially on another server of the cluster)

In fact, thinking more about this, monotonic reads and causal consistency are a requirement (not just a nice to have) even outside the HA scenario. It's just very easy to guarantee, assuming that writes are written to disk before returning (so if it restarts monotonic reads will be guaranteed)

However, write quorums only applies in the HA scenario of course (it doesn't apply if you have just one server)

[ylorph]

Thanks for the explanation,

• Causally consistent
  -> is what I meant by the ability to read in commit order , but this term is more correct indeed !
• Monotonic Reads
  -> I have difficulty with this one if A & B live on different streams and the stream B lives in has some TTL for the events or was deleted.

[skleanthous]

Monotonic reads doesn't necessarily refer to events, but changes. I just gave the above as an example. The "essence" of monotonic reads is basically always seeing changes going forward in time, and you should never go into a global state which was previous compared to earlier reads.

Let's see another example. Assume that two actions happened:

1. Balance changed of account 321 was changed to $10
2. Balance changed of account 321 was changed to $40

Monotonic reads would guarantee that in the absence of other changes, if a query read $40, and then a subsequent read would never get $10 because a node didn't get updated yet.

This extends even to streams. Let's assume a stream account-321. If two events get persisted due to two different operations:

1. Account initialized with balance $10
2. Deposit made, with value $30, and total balance $40

Then whatever event store instance we query the stream, a similar guarantee as the above example should be met. We should never get both events on a first query, with a second query returning just one event. Again, this is assuming an absence of changes.

The above guarantee is important even in single-instance servers. If writes are not hard (don't get flushed to disk before an event append request returns), then a restart of the server at the right time (for example) could result to a loss of the event so subsequent requests wouldn't see the next event. Most often though this guarantee is important to server clusters where it applies to the runnign system.

[ylorph]

https://jepsen.io/consistency/models/monotonic-reads
https://jepsen.io/consistency

https://blog.acolyer.org/2014/11/07/highly-available-transactions-virtues-and-limitations/

http://www.vldb.org/pvldb/vol7/p181-bailis.pdf
https://arxiv.org/pdf/1512.00168.pdf