Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

client: defensive against getting stale alloc updates #5906

Merged
merged 1 commit into from
Jul 2, 2019
Merged

client: defensive against getting stale alloc updates #5906

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions client/client.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1767,6 +1767,9 @@ func (c *Client) allocSync() {
// allocUpdates holds the results of receiving updated allocations from the
// servers.
type allocUpdates struct {
// index is index of server store snapshot used for fetching alloc status
index uint64

// pulled is the set of allocations that were downloaded from the servers.
pulled map[string]*structs.Allocation

Expand Down Expand Up @@ -1944,6 +1947,7 @@ OUTER:
filtered: filtered,
pulled: pulledAllocs,
migrateTokens: resp.MigrateTokens,
index: resp.Index,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

L1942 updates req.MinQueryIndex if and only if resp.Index is greater, so I wonder if there's some reason we should use req.MinQueryIndex here instead. I'm honestly not sure L1942 is reachable. Perhaps there's a timeout that could cause a response before resp.Index is greater than MinQueryIndex?

Not a blocker as I think at worst it's an edge case of an edge case that when hit will negate the correctness improvement of this PR. It can't make the behavior worse than before the PR AFAICT.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should be using req.MinQueryIndex here. It's simpler to reason that we reconciling local state against pulled state (at index resp.Index) without worrying much about the indirection or interference of req.MinQueryIndex` (i.e. if resp.Index is earlier than req.MinQueryIndex, using req.MinQueryIndex risks us believing the server state is more recent than it actually is). We expect the reconciler to work even if resp.Index went back in time expectedly.

As for req.MinQueryIndex, it seems that we are protecting against servers state going back in time! That feels quite odd and I wonder if it's just being defensive or a case we hit at some point.

}

select {
Expand Down
2 changes: 1 addition & 1 deletion client/util.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ func diffAllocs(existing map[string]uint64, allocs *allocUpdates) *diffResult {
_, filtered := allocs.filtered[existID]

// If not updated or filtered, removed
if !pulled && !filtered {
if !pulled && !filtered && allocs.index > existIndex {
result.removed = append(result.removed, existID)
continue
}
Expand Down