Make server logic vector clock-aware #153
Conversation
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @richardhuaaa and the rest of your teammates on |
Your org has enabled the Graphite merge queue for merging into mainAdd the label “Queue” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “Hotfix” to add to the merge queue as a hot fix. You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link. |
| type PollableDBQuery[ValueType any, CursorType any] func( | ||
| ctx context.Context, | ||
| lastSeen CursorType, | ||
| numRows int32, | ||
| ) (results []ValueType, nextCursor CursorType, err error) |
There was a problem hiding this comment.
Main change is here - PollableDBQuery now uses a generic cursor type rather than an int64
| @@ -16,7 +16,8 @@ import ( | |||
| ) | |||
|
|
|||
| const ( | |||
| maxRequestedRows int32 = 1000 | |||
| maxRequestedRows uint32 = 1000 | |||
| maxVectorClockLength int = 100 | |||
There was a problem hiding this comment.
I added this mainly to stop malicious actors from flooding our DB with large requests using huge vector clock queries. We could theoretically make this bigger
There was a problem hiding this comment.
In practice, this should always be < the number of nodes in the registry. Maybe that can be the upper bound?
I was worried about type-conversions between int32 and uint32 and overflow issues, but it looks like with two's complement, overflowed int32's behave the same as uint32's after conversion (or in other words, the increment operation does exactly the same thing to the underlying byte representation either way). In other words, our postgres schema can only support signed int32's, but it seems like we can just pretend they are uint32's and have the same effect.
We do still need to worry about the postgres sequence overflowing, as well as the max value of uint32/uint64 - when implementing the node-to-node syncing logic, my plan is to regularly log the node's vector clock, so that we can see it coming from a long way away.
Closes #132