Introduce group checksums #2964
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… the tablets in group changed.
…the values. Also, have a way to find violation linearly.
…that commit when Zero stops being a leader.
golangcibot
reviewed
Feb 1, 2019
dgraph/cmd/debug/run.go
Outdated
| for { | ||
| min, max := getMinMax(db, readTs-1) | ||
| if max <= min { | ||
| fmt.Println("Can't find it. Max: %d\n", max) |
There was a problem hiding this comment.
Println call has possible formatting directive %d (from govet)
dgraph/cmd/debug/run.go
Outdated
| @@ -66,7 +67,8 @@ func init() { | |||
| flag := Debug.Cmd.Flags() | |||
| flag.BoolVar(&opt.itemMeta, "item", true, "Output item meta as well. Set to false for diffs.") | |||
| flag.BoolVar(&opt.vals, "vals", false, "Output values along with keys.") | |||
| flag.BoolVar(&opt.jepsen, "jepsen", false, "Disect Jepsen output.") | |||
| flag.BoolVar(&opt.noKeys, "nokeys", false, "Ignore key_. Only consider amount_ when calculating total.") | |||
manishrjain
added a commit
that referenced
this pull request
Feb 1, 2019
This PR introduces group checksums, which get propagated along the OracleDeltas streaming from Zero -> Alpha Leader -> Alpha Followers. This allows an Alpha to block a read if the group checksum does not match the checksum in membership, which is being received independently directly from Zero. This fixes the Jepsen issue where during a tablet move, an Alpha with an older membership status (thinking it is serving the group) can reply to a query with stale data. Now, the Alpha would see that the checksum is different between Oracle Deltas and Membership state, and would block until it gets an updated state. Thus, it would realize that it is no longer serving the tablet, and either return an error or shoot the query off to the right Alpha. Proposal retry issue: This PR also fixes another issue where a commit proposal to Zero Leader gets blocked when leader goes under partition but before it steps down. When the proposal times out and is internally retried, that Zero is no longer the leader, causing `n.AmLeader()` to fail and the commit to get rejected by Zero. This moves the needle of MaxAssigned, allowing a read to happen at Alpha. But, the proposal which had timed out, does get applied after -- causing a commit to happen at a lower timestamp than MaxAssigned. This PR moves the `n.AmLeader` check to the top of the function, so when we are in the proposal try loop, we don't do this check. Instead, we'll keep trying to propose until we have a resolution from Raft. Debugging: This PR updates the `dgraph debug` tool to sum up the amounts, without looking at the keys in Jepsen. This is useful when keys are located separately from amounts in two different groups. This change is what allowed me to find the first txn violation, and the causes for tablet move nemesis. Fixes #2321 . Commits: * Introduce Group checksums, so an Alpha can know if the composition of the tablets in group changed. * Add a simple waiting loop for group checksums. * Remove any background cleaning jobs. * Always iterate to create a posting list. * Have a way to find violation linearly. * Make reading amounts and total without considering keys. * Avoid a race cond between proposing a commit in a loop v/s rejecting that commit when Zero stops being a leader. * Self review. Do not do any tablet removals or deletions for now.
dna2github
pushed a commit
to dna2fork/dgraph
that referenced
this pull request
Jul 19, 2019
This PR introduces group checksums, which get propagated along the OracleDeltas streaming from Zero -> Alpha Leader -> Alpha Followers. This allows an Alpha to block a read if the group checksum does not match the checksum in membership, which is being received independently directly from Zero. This fixes the Jepsen issue where during a tablet move, an Alpha with an older membership status (thinking it is serving the group) can reply to a query with stale data. Now, the Alpha would see that the checksum is different between Oracle Deltas and Membership state, and would block until it gets an updated state. Thus, it would realize that it is no longer serving the tablet, and either return an error or shoot the query off to the right Alpha. Proposal retry issue: This PR also fixes another issue where a commit proposal to Zero Leader gets blocked when leader goes under partition but before it steps down. When the proposal times out and is internally retried, that Zero is no longer the leader, causing `n.AmLeader()` to fail and the commit to get rejected by Zero. This moves the needle of MaxAssigned, allowing a read to happen at Alpha. But, the proposal which had timed out, does get applied after -- causing a commit to happen at a lower timestamp than MaxAssigned. This PR moves the `n.AmLeader` check to the top of the function, so when we are in the proposal try loop, we don't do this check. Instead, we'll keep trying to propose until we have a resolution from Raft. Debugging: This PR updates the `dgraph debug` tool to sum up the amounts, without looking at the keys in Jepsen. This is useful when keys are located separately from amounts in two different groups. This change is what allowed me to find the first txn violation, and the causes for tablet move nemesis. Fixes dgraph-io#2321 . Commits: * Introduce Group checksums, so an Alpha can know if the composition of the tablets in group changed. * Add a simple waiting loop for group checksums. * Remove any background cleaning jobs. * Always iterate to create a posting list. * Have a way to find violation linearly. * Make reading amounts and total without considering keys. * Avoid a race cond between proposing a commit in a loop v/s rejecting that commit when Zero stops being a leader. * Self review. Do not do any tablet removals or deletions for now.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces group checksums, which get propagated along the OracleDeltas streaming from Zero -> Alpha Leader -> Alpha Followers. This allows an Alpha to block a read if the group checksum does not match the checksum in membership, which is being received independently directly from Zero.
This fixes the Jepsen issue where during a tablet move, an Alpha with an older membership status (thinking it is serving the group) can reply to a query with stale data. Now, the Alpha would see that the checksum is different between Oracle Deltas and Membership state, and would block until it gets an updated state. Thus, it would realize that it is no longer serving the tablet, and either return an error or shoot the query off to the right Alpha.
Proposal retry issue:
This PR also fixes another issue where a commit proposal to Zero Leader gets blocked when leader goes under partition but before it steps down. When the proposal times out and is internally retried, that Zero is no longer the leader, causing
n.AmLeader()to fail and the commit to get rejected by Zero. This moves the needle of MaxAssigned, allowing a read to happen at Alpha. But, the proposal which had timed out, does get applied after -- causing a commit to happen at a lower timestamp than MaxAssigned.This PR moves the
n.AmLeadercheck to the top of the function, so when we are in the proposal try loop, we don't do this check. Instead, we'll keep trying to propose until we have a resolution from Raft.Debugging:
This PR updates the
dgraph debugtool to sum up the amounts, without looking at the keys in Jepsen. This is useful when keys are located separately from amounts in two different groups. This change is what allowed me to find the first txn violation, and the causes for tablet move nemesis.Fixes #2321 .
This change is