Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize node encoding/decoding with stream-vbyte format #673

Closed
wants to merge 2 commits into from

Conversation

yihuang
Copy link
Collaborator

@yihuang yihuang commented Feb 2, 2023

This is a node storage format change, need to migrate database, but I guess it's ok in light of new node key format, we'll need to migrate anyway, the same technique can be applied to new node key format as well.
The improvement is pretty good, we don't even use the SIMP optimizations, since we don't have that much integers.

Downside:

  • don't support version bigger than math.MaxUint32.
    • I did a calculation, say block time is 2 second, (((1<<32)-1) * 2) / 3600/24/365 = 272, it means it takes 272 years for the version number to overflow uint32, we are using 6 seconds block time right now, which makes it even longer.

Benchmark

$ go test -run=^$ -bench="maxEncodedSize|Encode|Decode" -benchmem ./ -count=10

Compare with this version where I adjusted the test cases the same as this PR:

name                          old time/op    new time/op    delta
Node_maxEncodedSize-12          7.33ns ± 4%    0.24ns ± 3%  -96.67%  (p=0.000 n=10+9)
Node_Encode/small_integer-12     245ns ± 1%      55ns ± 1%  -77.62%  (p=0.000 n=10+10)
Node_Encode/large_integer-12     245ns ± 2%      55ns ± 0%  -77.64%  (p=0.000 n=10+9)
Node_Decode/small_number-12      169ns ± 1%      65ns ± 2%  -61.73%  (p=0.000 n=10+9)
Node_Decode/large_number-12      170ns ± 1%      65ns ± 1%  -61.92%  (p=0.000 n=10+9)

name                          old alloc/op   new alloc/op   delta
Node_maxEncodedSize-12           0.00B          0.00B          ~     (all equal)
Node_Encode/small_integer-12      160B ± 0%      112B ± 0%  -30.00%  (p=0.000 n=10+10)
Node_Encode/large_integer-12      160B ± 0%      112B ± 0%  -30.00%  (p=0.000 n=10+10)
Node_Decode/small_number-12       256B ± 0%      160B ± 0%  -37.50%  (p=0.000 n=10+10)
Node_Decode/large_number-12       256B ± 0%      160B ± 0%  -37.50%  (p=0.000 n=10+10)

name                          old allocs/op  new allocs/op  delta
Node_maxEncodedSize-12            0.00           0.00          ~     (all equal)
Node_Encode/small_integer-12      2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Node_Encode/large_integer-12      2.00 ± 0%      1.00 ± 0%  -50.00%  (p=0.000 n=10+10)
Node_Decode/small_number-12       4.00 ± 0%      1.00 ± 0%  -75.00%  (p=0.000 n=10+10)
Node_Decode/large_number-12       4.00 ± 0%      1.00 ± 0%  -75.00%  (p=0.000 n=10+10)

@yihuang yihuang requested a review from a team as a code owner February 2, 2023 07:35
@yihuang yihuang changed the title benchmark stream vbyte encoding Optimize node encoding/decoding with stream-vbyte format Feb 2, 2023
@yihuang yihuang marked this pull request as draft February 2, 2023 08:26
@yihuang
Copy link
Collaborator Author

yihuang commented Feb 8, 2023

I'll close it for now, no resource to pursue it for now.

@yihuang yihuang closed this Feb 8, 2023
@tac0turtle
Copy link
Member

Would love to open an issue for this. It looks awesome, maybe someone from outside can pick it up

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants