You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As part of #6363, we'd like the swing-store snapStore to have an API where you read a snapshot by getting a stream (an iterator of Uint8Arrays), instead of running an async thunk against a temporary file on disk. Likewise, we'd like an API where you can write a snapshot into the DB by passing in a stream (anything that iterates Uint8Arrays).
The existing tempfile-based APIs are named loadSnapshot and saveSnapshot.
I'm thinking that we should add stream-based readSnapshot and writeSnapshot APIs, in parallel with the old APIs, to reduce the flag-day coordination with #6363 (since we can't stop using the tempfile APIs until xsnap's C code is changed to accept/produce streams). I think the #6363 plan might include doing adaptation on the kernel side (so xsnap does a stream to the kernel, but the kernel-side manager writes the stream into the snapStore tempfile). That would reduce the couping further.
Reduces our disk space footprint a bit. Memory footprint should remain the same. Should reduce latency of snapshot operations, in that it enables parallel execution of the kernel/DB code and the xsnap worker, in separate CPU cores, instead of having a synchronization/lock-step point between the two sides.
Test Plan
Unit tests for the new APIs.
The text was updated successfully, but these errors were encountered:
As part of the preliminary work for #6363 I will be doing a variation of what is proposed above: the xsnap interface will accept / produce streams and internally manage the temporary file, and thus I will have to update the snapStore interface to deal with these streams instead of filePath. Whether to get rid of the callback model or not is TBD, but it will likely simplify things further.
What is the Problem Being Solved?
As part of #6363, we'd like the swing-store
snapStore
to have an API where you read a snapshot by getting a stream (an iterator ofUint8Array
s), instead of running an async thunk against a temporary file on disk. Likewise, we'd like an API where you can write a snapshot into the DB by passing in a stream (anything that iterates Uint8Arrays).The existing tempfile-based APIs are named
loadSnapshot
andsaveSnapshot
.I'm thinking that we should add stream-based
readSnapshot
andwriteSnapshot
APIs, in parallel with the old APIs, to reduce the flag-day coordination with #6363 (since we can't stop using the tempfile APIs until xsnap's C code is changed to accept/produce streams). I think the #6363 plan might include doing adaptation on the kernel side (so xsnap does a stream to the kernel, but the kernel-side manager writes the stream into the snapStore tempfile). That would reduce the couping further.Description of the Design
snapStore.readSnapshot(vatID) -> Iterable<Uint8Array>
snapStore.writeSnapshot(vatID, endPos, Iterable<Uint8Array>) -> Promise(SnapshotResult)
Security Considerations
Shouldn't be any.
Scaling Considerations
Reduces our disk space footprint a bit. Memory footprint should remain the same. Should reduce latency of snapshot operations, in that it enables parallel execution of the kernel/DB code and the xsnap worker, in separate CPU cores, instead of having a synchronization/lock-step point between the two sides.
Test Plan
Unit tests for the new APIs.
The text was updated successfully, but these errors were encountered: