Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix OrderByOperator.finish to wait for finishMemoryRevoke #16431

Conversation

findepi
Copy link
Member

@findepi findepi commented Mar 8, 2023

The Operator.startMemoryRevoke API promises that no processing methods
are called before memory revoking is done (finishMemoryRevoke). Since
8c9df82, 7d11573 and cf50577 finish is the only processing
method that can be called between startMemoryRevoke and
finishMemoryRevoke.

OrderByOperator had a concurrency-like bug because it did not handle
well the case when invocation order is startMemoryRevoke, spill
completes, finish, finishMemoryRevoke. During finish the operator
gets an iterator over PagesIndex which becomes invalidated when
finishMemoryRevoke clears the PagesIndex.

The commit fixes that by adding explicit wait for finishMemoryRevoke
inside OrderByOperator.finish. The check is same as in
HashBuilderOperator. For consistency with the other operator the
finishMemoryRevoke field becomes Optional.

Fixes #16406

@@ -92,6 +93,7 @@
private final IntArrayList positionCounts;
private final boolean eagerCompact;

private int modificationCount; // may overflow, doesn't matter
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

volatile?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i don't think PagesIndex is supposed to be thread safe?

@findepi
Copy link
Member Author

findepi commented Mar 8, 2023

cc @pettyjamesm

findepi added 2 commits March 8, 2023 13:31
The `Operator.startMemoryRevoke` API promises that no processing methods
are called before memory revoking is done (`finishMemoryRevoke`).  Since
8c9df82,
7d11573 and
cf50577 `finish` is the only processing
method that can be called between `startMemoryRevoke` and
`finishMemoryRevoke`.

`OrderByOperator` had a concurrency-like bug because it did not handle
well the case when invocation order is `startMemoryRevoke`, spill
completes, `finish`, `finishMemoryRevoke`. During `finish` the operator
gets an iterator over `PagesIndex` which becomes invalidated when
`finishMemoryRevoke` clears the `PagesIndex`.

The commit fixes that by adding explicit wait for `finishMemoryRevoke`
inside `OrderByOperator.finish`. The check is same as in
`HashBuilderOperator`. For consistency with the other operator the
`finishMemoryRevoke` field becomes `Optional`.
When `PagesIndex` is mutated, the iteration behavior is not well
defined. Similarly to JDK collections, fail iteration with
`ConcurrentModificationException` when underlying index has changed.
Unlike the JDK collections, the check is done only at the end (not
fail-fast), to minimize performance impact.

Before recent fixes, `OrderByOperator` would create an iterator over an
index and then clear it. This was not intentional and led to a
regression when `PagesIndex` iteration code was refactored. The check
introduced here would catch that situation explicitly.
@findepi findepi force-pushed the findepi/fix-orderbyoperator-finish-to-wait-for-finishmemoryrevoke-85d741 branch from ab69353 to 45989e4 Compare March 8, 2023 12:40
Copy link
Member

@sopel39 sopel39 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is test possible?

@findepi
Copy link
Member Author

findepi commented Mar 8, 2023

is test possible?

the change is nicely tested with TestSpilledOrderByQueries which, before the change, fails for me every single time i run it

@findepi
Copy link
Member Author

findepi commented Mar 8, 2023

CI #16437

Copy link
Member

@pettyjamesm pettyjamesm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, although +1 to the part about adding a comment and one question about returning early vs finishing revoking and proceeding.

@findepi findepi merged commit 1d908ea into trinodb:master Mar 8, 2023
@findepi findepi deleted the findepi/fix-orderbyoperator-finish-to-wait-for-finishmemoryrevoke-85d741 branch March 8, 2023 19:06
@github-actions github-actions bot added this to the 410 milestone Mar 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

Potential regression in spilled ORDER BY
5 participants