Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: deflake node drain intergration test #18171

Merged
merged 1 commit into from
Aug 14, 2023
Merged

Conversation

tgross
Copy link
Member

@tgross tgross commented Aug 7, 2023

The TestDrainer_AllTypes_NoDeadline test has been flaky. It looks like this might be because the final update of batch allocations to complete is improperly updating the state store directly rather than by RPC. If the service jobs have restarted in the meantime, the allocClientStateSimulator will have updated the index on the allocations table and that will prevent the drainer from unblocking (and being marked complete) when the batch jobs are written with an earlier index.

This changeset attempts to fix that by making the update via RPC (as it normally would be in real code).

Fixes: #18147

The `TestDrainer_AllTypes_NoDeadline` test has been flaky. It looks like this
might be because the final update of batch allocations to complete is improperly
updating the state store directly rather than by RPC. If the service jobs have
restarted in the meantime, the `allocClientStateSimulator` will have updated the
index on the allocations table and that will prevent the drainer from
unblocking (and being marked complete) when the batch jobs are written with an
earlier index.

This changeset attempts to fix that by making the update via RPC (as it normally
would be in real code).
Copy link
Contributor

@shoenig shoenig left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@tgross tgross merged commit ac8604e into main Aug 14, 2023
@tgross tgross deleted the flaky-test-drain-no-deadline branch August 14, 2023 20:17
tgross added a commit that referenced this pull request Aug 14, 2023
The `TestDrainer_AllTypes_NoDeadline` test has been flaky. It looks like this
might be because the final update of batch allocations to complete is improperly
updating the state store directly rather than by RPC. If the service jobs have
restarted in the meantime, the `allocClientStateSimulator` will have updated the
index on the allocations table and that will prevent the drainer from
unblocking (and being marked complete) when the batch jobs are written with an
earlier index.

This changeset attempts to fix that by making the update via RPC (as it normally
would be in real code).
tgross added a commit that referenced this pull request Aug 14, 2023
The `TestDrainer_AllTypes_NoDeadline` test has been flaky. It looks like this
might be because the final update of batch allocations to complete is improperly
updating the state store directly rather than by RPC. If the service jobs have
restarted in the meantime, the `allocClientStateSimulator` will have updated the
index on the allocations table and that will prevent the drainer from
unblocking (and being marked complete) when the batch jobs are written with an
earlier index.

This changeset attempts to fix that by making the update via RPC (as it normally
would be in real code).
tgross added a commit that referenced this pull request Aug 14, 2023
The `TestDrainer_AllTypes_NoDeadline` test has been flaky. It looks like this
might be because the final update of batch allocations to complete is improperly
updating the state store directly rather than by RPC. If the service jobs have
restarted in the meantime, the `allocClientStateSimulator` will have updated the
index on the allocations table and that will prevent the drainer from
unblocking (and being marked complete) when the batch jobs are written with an
earlier index.

This changeset attempts to fix that by making the update via RPC (as it normally
would be in real code).
@tgross
Copy link
Member Author

tgross commented Aug 14, 2023

Backported to 1.6.x, 1.5.x, and 1.4.x

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

test flake: TestDrainer_AllTypes_NoDeadline
2 participants