Merge #36554

36554: roachpb: formalize observed timestamp preconditions, talk about leaseholders r=nvanbenschoten a=nvanbenschoten

This PR is an initial reaction to #36431 and to `limitTxnMaxTimestamp` being
broken for follower reads (PR for that coming soon). To start, we were missing
any real documentation on what makes the observed timestamp mechanism safe and
what invariants need to hold in order for a clock reading on a node to imply an
absence of causality on any values later read at higher timestamp on that node.
The PR addresses this by discussing the invariants necessary for observed
timestamps to be used to bound a transaction's uncertainty window.

In doing so, the PR also attempts to address a second issue, which is that the
documentation conflated a node serving reads and writes with leaseholders
serving reads and writes. At the time that this was all written, these were
equivalent. Now that we have follower reads, the omission of a discussion about
leaseholders made this all hard to think about (e.g. "whose observed timestamp
do we care about when performing a follower read?"). The PR addresses this by
adjusting the wording around nodes and leaseholders. This would have come in
handy in reasoning about #23749.

Release note: None

Co-authored-by: Nathan VanBenschoten <[email protected]>