-
Notifications
You must be signed in to change notification settings - Fork 173
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: introduce initial snapshot chunking #2119
Conversation
479c877
to
f2b9b34
Compare
f2b9b34
to
e302409
Compare
e302409
to
1ebf667
Compare
✅ Deploy Preview for electric-next ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
c531cfc
to
4d12065
Compare
4d12065
to
e4a1074
Compare
@msfstef can you please take another look at |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very nice work! Love the ripping out of snapshot logic in the serve plug!
Left some comments and questions - as for the TS tests checking etags, we can modify them to be looking at the first "incomplete" actual log chunk rather than looking at -1
(which used to collate the transaction log)
101a968
to
1a28bea
Compare
This PR changes the way we write snapshots to disk, the storage API to request them, and some logic around offsets.
Storage.get_snapshot/1
method is gone. It's now unified underStorage.get_log_stream/3
, with some caveats.offset
field to theelectric-offset
header of the same response. All our clients respect the header, and that's the only correct behaviour going forward. There is a decision to removeoffset
from our items entirely, but that's a separate PR.LogOffset.new(0, :infinity)
, and consider all offsets at txn 0 to be virtual pointers.get_log_stream/3
for the snapshot section ignores the contract a little bit - you cannot request the stream to be up to an arbitrary point, it's always up to the chunk boundary, even when specified otherwise. Since we've moved to "pure" chunking behaviour, it should be enshrined in theStorage
interface so that reading functions can act optimally and honestly. I'm hoping to address that in a separate PR