Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Save storage improvements #39

Closed
mhofman opened this issue Dec 7, 2021 · 1 comment · Fixed by #44
Closed

Save storage improvements #39

mhofman opened this issue Dec 7, 2021 · 1 comment · Fixed by #44
Assignees
@mhofman
Labels
telemetry

Comments

@mhofman
Copy link
Member

mhofman commented Dec 7, 2021

Currently the loadgen runner can be configured to take a chain storage snapshot for each stage, which can get pretty slow.

Here are a few ideas of improvements:

  • Make a copy of the storage folder and compress it async while starting the next stage. Would be even better if this was on a copy-on-write fs
  • Compress with a less expensive algo (currently xz) or not at all
  • Automatically snapshot the storage if a stage fails, even if no-storage.
@mhofman mhofman self-assigned this Dec 7, 2021
@mhofman
Copy link
Member Author

mhofman commented Dec 7, 2021

Also:

  • Save client / solo storage
  • Save ag-solo Slog

@mhofman mhofman mentioned this issue Dec 7, 2021
Closed
@mhofman mhofman mentioned this issue Jan 8, 2022
Merged
mhofman added a commit that referenced this issue Jan 17, 2022
@mhofman
Improve runner compatibility with testnet (#44)
376964d 
- Update to node 16 and Debian bullseye (with fallback to node 14 for older incompatible SDKs)
- Handle some older SDK versions which output lockdown sniffing to stdout instead of stderr
- Rewrote the config argv parsing logic, to make it behave slightly more sanely
- Fixed some deadlock issues, e.g. adding some timeouts on task ready (see #40) or slog streams not closing properly
- Capture client storage and slog file (should help track some transient seg faults in the solo)
- Automatically capture the state of the client and chain if an error occurs (see #39)
- Background the compression of the state directories after snapshotting them (overlayfs supports CoW). Closes #39 
- Avoid resetting the whole `agoric-servers` project in `local-chain` tests. Removes loadgen project `git` dependency.
- Elide long lines from the chain or solo output (improves github actions perf, see Agoric/agoric-sdk#4113)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mhofman mhofman

Labels
telemetry
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Better runner compatibility with testnet Agoric/testnet-load-generator
1 participant
@mhofman