Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test new approaches for persisting caches to disk #365

Closed
whitfin opened this issue Sep 13, 2024 · 1 comment · Fixed by #369
Closed

Test new approaches for persisting caches to disk #365

whitfin opened this issue Sep 13, 2024 · 1 comment · Fixed by #369
Assignees
Milestone

Comments

@whitfin
Copy link
Owner

whitfin commented Sep 13, 2024

The current approach requires loading a full cache and exporting an entire cache via Erlang terms. We should see if we can do something to improve this, such as streamed input.

If this isn't possible, it's not the end of the world, just want to take a look and see if we can do better.

@whitfin whitfin added this to the v4.0.0 milestone Sep 13, 2024
@whitfin
Copy link
Owner Author

whitfin commented Sep 13, 2024

I tested some approaches, and I think I will do a lot of changes here for v4.x.

  • Replace Cachex.export/2, simply delegate to Cachex.stream/3
  • Replace Cachex.dump/3 with a batched implementation using Cachex.stream/3
  • Replace Cachex.load/3 with a streaming implementation (consumed by Cachex.import/3)

I think Cachex.import/3 can stay as is, although it should accept a stream of records as well. There's potential to simply remove Cachex.export/2 but keeping it as a grace function is probably fine.

The result of all of this is that we're reusing common foundations in the implementations, and we're much lower on memory. Performance is a little slower, but we gain memory safety and we also get the ability to export based on a query, which is pretty cool.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant