Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce unnecessary snapshots by storing local snapshot metadata #427

Open
marcua opened this issue Aug 11, 2024 · 0 comments
Open

Reduce unnecessary snapshots by storing local snapshot metadata #427

marcua opened this issue Aug 11, 2024 · 0 comments

Comments

@marcua
Copy link
Owner

marcua commented Aug 11, 2024

  • Right now we dump a database on each periodic snapshot run
  • We then hash the database, and if the hash exists in S3-compatible storage, we don't upload it
  • We can avoid having to snapshot/hash by storing ephemeral metadata on the last time we ran snapshotting: if no file was modified since our last snapshot run, don't dump the database/hash it at al. More technically, get fs::metadata of each file in the dir, call modified() on result, sort the times so it's stable, take max or hash them together. If the max/hash of the modified times hasn't changed, skip this backup.
@marcua marcua converted this from a draft issue Aug 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: To do
Development

No branches or pull requests

1 participant