fixes #1920 #2049
fixes #1920 #2049
Conversation
ivakegg
commented
Jul 28, 2023
ivakegg
commented
- Added comment explaining why we do not persist the ivarator set in some situations.
| // Note that we will not persist the data if we are not over the scan threshold and the data fits into | ||
| // one buffer (no persisted data). That means that if we need to rebuild the sorted set that we can do | ||
| // it relatively quickly so no need for a persisted set with a complete marker. |
There was a problem hiding this comment.
I'm not sure what this comment has to do with the original issue I reported.
In the scenario I described, one of two things happen.
Either,
- We have entries buffered in memory, and that triggers the ivarator to persist the entries to disk and create a
completefile.
OR
- We have no entries buffered in memory (because everything was ALREADY persisted to disk), and that does NOT trigger the creation of a
completefile.
I don't think this has anything to do with the scan threshold. If it did, then in scenario #1 the code wouldn't persist the buffer to disk, (but it absolutely does!!!). Scenario #1 is not the problem.
Scenario #2 where data has already been written to disk, but there are no results in the in-memory buffer is the problem. We already have data written to disk, but we aren't creating the complete file.
There was a problem hiding this comment.
I am pretty sure that "this.set.hasPersistedData()" is the part of the check that takes care of your scenario 2.
There was a problem hiding this comment.
@jwomeara are you saying that the code in scenario 2 is not making an accurate determination about the sate of persistence? That it's not enough that the buffer is null to call it "persisted". Should the check be buffer == null && isPersisted instead of an || ? Or am I misunderstanding altogether?
