Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add streaming readSnapshot API to snapStore #7490

Closed
warner opened this issue Apr 24, 2023 · 1 comment · Fixed by #7531
Closed

add streaming readSnapshot API to snapStore #7490

warner opened this issue Apr 24, 2023 · 1 comment · Fixed by #7531
Assignees
Labels
enhancement New feature or request SwingSet package: SwingSet vaults_triage DO NOT USE

Comments

@warner
Copy link
Member

warner commented Apr 24, 2023

What is the Problem Being Solved?

As part of #6363, we'd like the swing-store snapStore to have an API where you read a snapshot by getting a stream (an iterator of Uint8Arrays), instead of running an async thunk against a temporary file on disk. Likewise, we'd like an API where you can write a snapshot into the DB by passing in a stream (anything that iterates Uint8Arrays).

The existing tempfile-based APIs are named loadSnapshot and saveSnapshot.

I'm thinking that we should add stream-based readSnapshot and writeSnapshot APIs, in parallel with the old APIs, to reduce the flag-day coordination with #6363 (since we can't stop using the tempfile APIs until xsnap's C code is changed to accept/produce streams). I think the #6363 plan might include doing adaptation on the kernel side (so xsnap does a stream to the kernel, but the kernel-side manager writes the stream into the snapStore tempfile). That would reduce the couping further.

Description of the Design

  • snapStore.readSnapshot(vatID) -> Iterable<Uint8Array>
  • snapStore.writeSnapshot(vatID, endPos, Iterable<Uint8Array>) -> Promise(SnapshotResult)

Security Considerations

Shouldn't be any.

Scaling Considerations

Reduces our disk space footprint a bit. Memory footprint should remain the same. Should reduce latency of snapshot operations, in that it enables parallel execution of the kernel/DB code and the xsnap worker, in separate CPU cores, instead of having a synchronization/lock-step point between the two sides.

Test Plan

Unit tests for the new APIs.

@warner warner added enhancement New feature or request SwingSet package: SwingSet labels Apr 24, 2023
@mhofman mhofman self-assigned this Apr 24, 2023
@mhofman
Copy link
Member

mhofman commented Apr 24, 2023

As part of the preliminary work for #6363 I will be doing a variation of what is proposed above: the xsnap interface will accept / produce streams and internally manage the temporary file, and thus I will have to update the snapStore interface to deal with these streams instead of filePath. Whether to get rid of the callback model or not is TBD, but it will likely simplify things further.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request SwingSet package: SwingSet vaults_triage DO NOT USE
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants