Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compute disk buffer initial_size with Jean's method #7408

Closed
blt opened this issue May 11, 2021 · 2 comments
Closed

Compute disk buffer initial_size with Jean's method #7408

blt opened this issue May 11, 2021 · 2 comments
Assignees
Labels
domain: buffers Anything related to Vector's memory/disk buffers

Comments

@blt
Copy link
Contributor

blt commented May 11, 2021

@blt Actually, looking at the crate, there does appear to be a LevelDBIterator trait, which has a from (and to) method which seems to imply we can iterate over an arbitrary range of keys.

It seems to me we could create a DB handle, iterate X times, get the key of the last item, and set that as the starting key for the next DB handle iterator. Reading the offiial docs it appears key ordering happens lexicographically, which implies we can have stable ordering across DB handles at startup when the cache isn't actually in use, correct?

Originally posted by @JeanMertz in #7398 (comment)

@blt blt self-assigned this May 11, 2021
@blt blt added the domain: buffers Anything related to Vector's memory/disk buffers label May 11, 2021
@ktff
Copy link
Contributor

ktff commented May 12, 2021

This should solve the OOM issue reported in #6741 (comment).

cc. @mikhno-s

@blt
Copy link
Contributor Author

blt commented May 13, 2021

We gave this method a shot in #7410 but the startup cost imposed by the technique is simply too high to be useful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
domain: buffers Anything related to Vector's memory/disk buffers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants