kvclient/rangefeed: make frontier advances observable using an option #71256
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This change is |
arulajmani
force-pushed
the
rangefeed-on-frontier-advance
branch
2 times, most recently
from
19b5d2c to
44752c4
Compare
irfansharif
approved these changes
Oct 7, 2021
|
@ajwerner or @nvanbenschoten does one of you want to take a second look at this as well or is this good to go once Irfan's comments are addressed? |
arulajmani
force-pushed
the
rangefeed-on-frontier-advance
branch
from
44752c4 to
4eab4e6
Compare
ajwerner
approved these changes
Oct 14, 2021
There was a problem hiding this comment.
Reviewed 2 of 3 files at r1.
Reviewable status:complete! 1 of 0 LGTMs obtained (waiting on @arulajmani and @nvanbenschoten)
pkg/kv/kvclient/rangefeed/rangefeed_external_test.go, line 198 at r2 (raw file):
minTS.Backward(ts) } require.True(
assert.True because it's a separate goroutine?
pkg/kv/kvclient/rangefeed/rangefeed_external_test.go, line 198 at r2 (raw file):
minTS.Backward(ts) } require.True(
Truef?
This patch adds a `OnFrontierAdvance` option to the rangefeed libary. This option allows users of this library to install a callback which is called whenever the frontier is advanced (with the new frontier timestamp). The frontier is used to track the minimum resolvedTS accross all ranges that the rangefeed span breaks down to. The intention is to allow users of this library to be able to use this option to checkpoint the rangefeed progress. The plan is to use it as such for cockroachdb#69661 and cockroachdb#69614. Release note: None
arulajmani
force-pushed
the
rangefeed-on-frontier-advance
branch
from
4eab4e6 to
fc803ee
Compare
arulajmani
commented
Oct 14, 2021
There was a problem hiding this comment.
TFTRs!
bors r+
Reviewable status:
pkg/kv/kvclient/rangefeed/rangefeed_external_test.go, line 198 at r2 (raw file):
Previously, ajwerner wrote…
Truef?
Done.
|
Build succeeded: |
craig bot
pushed a commit
that referenced
this pull request
Oct 18, 2021
71225: rangefeedbuffer: introduce a rangefeed buffer r=irfansharif a=irfansharif Buffer provides a thin memory-bounded buffer to sit on top of a rangefeed. It accumulates raw rangefeed events[^1], which can be flushed out in timestamp sorted order en-masse whenever the rangefeed frontier is bumped. If we accumulate more events than the limit allows for, we error out to the caller. We need such a thing in both #69614 and #69661. [^1]: Rangefeed error events are propagated to the caller, checkpoint events are discarded. Release note: None First commit is from #71256. Co-authored-by: Arul Ajmani <[email protected]>. 71534: ui/sql: show summarized statements in the statements table r=lindseyjin a=lindseyjin Resolves #27021 Previously, statements on the statements page hid too much information. There were complaints that it was difficult to disambiguate between statements without having to view the full query on the tooltips. The first commit in this patch implemented back-end changes to add a new metadata field for summarized queries, as well as formatting functions. This second commit implements additional logic to pass that new metadata to the front-end and display it in the Statements Table. Currently, we only create summaries of SELECT, INSERT/UPSERT, and UPDATE statements in the back-end. For all other statement types, we will continue to use the existing summary system.  Release note (ui change): Show new statement summaries on the Statements page. This applies for SELECT, INSERT/UPSERT, and UPDATE statements, and will enable them to be more detailed and less ambiguous than our previous formats. 71625: clusterversion: add a (disabled) assertion that binary version is latest r=dt a=dt This is intended to be flipped on and the release version updated when the cluster version mint commit is backported to a release branch. Doing so would then prevent accidentally backporting any future cluster versions without causing this to panic in all tests. Release note: none. Co-authored-by: irfan sharif <[email protected]> Co-authored-by: Lindsey Jin <[email protected]> Co-authored-by: David Taylor <[email protected]>
21 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This patch adds a
OnFrontierAdvanceoption to the rangefeed libary.This option allows users of this library to install a callback which
is called whenever the frontier is advanced.
The frontier is used to track the minimum resolvedTS accross all ranges
that the rangefeed span breaks down to. The intention is to allow users
of this library to be able to use this option to checkpoint the
rangefeed progress. The plan is to use it as such for #69661 and
#69614.
Release note: None