Persist block at retention edge when building index after peer streaming #1531
Conversation
Codecov Report
@@ Coverage Diff @@
## master #1531 +/- ##
======================================
Coverage 71.6% 71.6%
======================================
Files 948 948
Lines 77834 77834
======================================
Hits 55789 55789
Misses 18400 18400
Partials 3645 3645
Continue to review full report at Codecov.
|
| @@ -818,10 +832,12 @@ func (s *fileSystemSource) persistBootstrapIndexSegment( | |||
| // and we didn't bootstrap any more/less | |||
| requireFulfilled := expectedRanges.Copy() | |||
| requireFulfilled.Subtract(fulfilled) | |||
| exactStartEnd := min.Equal(blockStart) && max.Equal(blockStart.Add(blockSize)) | |||
| exactStartEnd := max.Equal(blockStart.Add(blockSize)) | |||
There was a problem hiding this comment.
not part of your change but the comment above is missing a "we" -> "Check that we completely..."
| blockEnd := blockStart.Add(blockSize) | ||
| expectedRangeStart, expectedRangeEnd := blockStart, blockEnd | ||
|
|
||
| // If basically the index block covers multiple data blocks |
There was a problem hiding this comment.
Can you make this more concrete and a little easier to understand:
Index blocks can be arbitrarily larger than data blocks, but the retention of the namespace is based on the size of the data blocks, not the index blocks. As a result, it's possible that the block start for the earliest index block is before the earliest possible retention time. If that is the case, then we snap the expected range start to the earliest retention block start because that is the point in time for which we'll actually have data available to construct indices from
Example Diagram:
Index block size: 4 hours
Data block size: 2 hours
Retention: 6 hours
[10AM -> 2PM][2PM -> 6PM] (Index Blocks)
[12PM->2PM][2PM->4PM][4PM->6PM] (Data Blocks)
I dont know if what I wrote is 100% accurate but you get the idea
There was a problem hiding this comment.
Also I understand how this resolves the validation check below where we make sure we've fulfilled the correct range, but how does it prevent the memory segment from being created? Also, the index block that starts before the retention begins is supposed to contain some data from the data block that it does overlap that IS within the retention, is that data correctly included in the index?
In the example above, the 10AM-2PM index block should contain the series that were in the 12PM-2PM data block, are we handling that correctly?
There was a problem hiding this comment.
It doesn't stop the memory segment being created, it just makes sure that it turns it into an FST segment instead (and hence it becomes a not-mutable segment).
Before it was failing validation and would just return it as the mem segment that was built.
There was a problem hiding this comment.
Yes we handle that case correctly.
There was a problem hiding this comment.
TY I used your diagram, appreciated!
No description provided.