Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pool Key lists and buffers in Pebble store #105

Merged
merged 1 commit into from
Oct 19, 2022
Merged

Conversation

masih
Copy link
Member

@masih masih commented Oct 16, 2022

Pool buffers, keys, keyer and key list in order to reduce CPU and memory footprint in Pebble store implementation.

Increase merger efficiency by avoiding recursion when flattening marshalled value keys. Instead, use mod to skip to where the serialized value keys should begin. This works because varint length is always added to the begining of value keys and their length is fixed in pebble store to 20 + 1.

During merge any value keys with unexpected key prefix are skipped. This automatically removes such keys from the store as they will be excluded from the merge.

store/pebble/codec.go Show resolved Hide resolved
store/pebble/key.go Show resolved Hide resolved
store/pebble/pool.go Outdated Show resolved Hide resolved
store/pebble/pool.go Outdated Show resolved Hide resolved
store/pebble/pool.go Outdated Show resolved Hide resolved
Pool buffers, keys, keyer and key list in order to reduce CPU and memory
footprint in Pebble store implementation.

Increase merger efficiency by avoiding recursion when flattening
marshalled value keys. Instead, use mod to skip to where the serialized
value keys should begin. This works because varint length is always
added to the begining of value keys and their length is fixed in pebble
store to 20 + 1.

During merge any value keys with unexpected key prefix are skipped.
This automatically removes such keys from the store as they will be
excluded from the merge.
@masih masih force-pushed the masih/pebble_codec_pooled branch from e30af36 to 0606c78 Compare October 19, 2022 13:35
@masih
Copy link
Member Author

masih commented Oct 19, 2022

Workload 0 benchmark for comparison:

$ go test ./bench -bench='(Pebble|Storethehash)(Put|Get)_W(0)'  -count=5 -timeout=300m 

BenchmarkStore_PebblePut_W0-8                  1        1988610008 ns/op          17.15 MB/s      48424081 size 201391560 B/op   4329444 allocs/op
--- BENCH: BenchmarkStore_PebblePut_W0-8
    store_bench_workloads_test.go:33: Workload 1 generated 1000 values with total size of 32.522 MiB.
BenchmarkStore_PebblePut_W0-8                  1        1686782294 ns/op          20.22 MB/s      48417771 size 195709632 B/op   4192982 allocs/op
BenchmarkStore_PebblePut_W0-8                  1        1643019567 ns/op          20.76 MB/s      48424081 size 195539184 B/op   4192796 allocs/op
BenchmarkStore_PebblePut_W0-8                  1        1693129651 ns/op          20.14 MB/s      48417771 size 193930168 B/op   4156038 allocs/op
BenchmarkStore_PebblePut_W0-8                  1        1706912678 ns/op          19.98 MB/s      48424081 size 194735544 B/op   4177132 allocs/op
BenchmarkStore_PebbleGet_W0-8                  1        6024332750 ns/op           5.66 MB/s      45466996 size 959914736 B/op  21814745 allocs/op
BenchmarkStore_PebbleGet_W0-8                  1        6013348175 ns/op           5.67 MB/s      45466996 size 959271576 B/op  21798756 allocs/op
BenchmarkStore_PebbleGet_W0-8                  1        6069595870 ns/op           5.62 MB/s      45466996 size 960972840 B/op  21835899 allocs/op
BenchmarkStore_PebbleGet_W0-8                  1        5990691385 ns/op           5.69 MB/s      45466996 size 960192128 B/op  21815468 allocs/op
BenchmarkStore_PebbleGet_W0-8                  1        6145057088 ns/op           5.55 MB/s      45466996 size 960327352 B/op  21821678 allocs/op
BenchmarkStore_StorethehashPut_W0-8            1        3722073469 ns/op           9.16 MB/s      87396304 size 973280640 B/op  11670663 allocs/op
BenchmarkStore_StorethehashPut_W0-8            1        3630226099 ns/op           9.39 MB/s      87392708 size 986805832 B/op  11704640 allocs/op
BenchmarkStore_StorethehashPut_W0-8            1        3680018004 ns/op           9.27 MB/s      87396126 size 989693960 B/op  11747880 allocs/op
BenchmarkStore_StorethehashPut_W0-8            1        3343566343 ns/op          10.20 MB/s      87378262 size 968908464 B/op  11908723 allocs/op
BenchmarkStore_StorethehashPut_W0-8            1        3314806176 ns/op          10.29 MB/s      87376078 size 970933888 B/op  11963848 allocs/op
BenchmarkStore_StorethehashGet_W0-8            1        9823852154 ns/op           3.47 MB/s      87374876 size 1778294872 B/op 32526981 allocs/op
--------------------------------------------------------
Store_PebblePut_W0-8           1.74s ±14%
Store_PebbleGet_W0-8           6.05s ± 2%
Store_StorethehashPut_W0-8     3.54s ± 6%
Store_StorethehashGet_W0-8     9.71s ± 4%

name                        speed
Store_PebblePut_W0-8        19.6MB/s ±13%
Store_PebbleGet_W0-8        5.64MB/s ± 2%
Store_StorethehashPut_W0-8  9.66MB/s ± 6%
Store_StorethehashGet_W0-8  3.51MB/s ± 4%

name                        size
Store_PebblePut_W0-8           48.4M ± 0%
Store_PebbleGet_W0-8           45.5M ± 0%
Store_StorethehashPut_W0-8     87.4M ± 0%
Store_StorethehashGet_W0-8     87.4M ± 0%

name                        alloc/op
Store_PebblePut_W0-8           196MB ± 3%
Store_PebbleGet_W0-8           960MB ± 0%
Store_StorethehashPut_W0-8     978MB ± 1%
Store_StorethehashGet_W0-8    1.75GB ± 3%

name                        allocs/op
Store_PebblePut_W0-8           4.21M ± 3%
Store_PebbleGet_W0-8           21.8M ± 0%
Store_StorethehashPut_W0-8     11.8M ± 1%
Store_StorethehashGet_W0-8     32.0M ± 3%

@masih masih marked this pull request as ready for review October 19, 2022 13:43
Copy link
Collaborator

@gammazero gammazero left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@masih masih merged commit 28d6b0b into main Oct 19, 2022
@masih masih deleted the masih/pebble_codec_pooled branch October 19, 2022 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants