Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: add fork-choice benchmark for updateHead #5577

Merged
merged 2 commits into from
May 31, 2023
Merged

test: add fork-choice benchmark for updateHead #5577

merged 2 commits into from
May 31, 2023

Conversation

dapplion
Copy link
Contributor

Motivation

Description

There's already a benchmark for compute deltas, add a second benchmark for updateHead. The times computed locally deviate significantly from what's observed in production.

600_000 network updateHead():

  • in benchmarks: 20 ms
  • in production: 100-200 ms

No pointers currently on what can cause such a disparity.

@dapplion dapplion requested a review from a team as a code owner May 30, 2023 09:06
@github-actions
Copy link
Contributor

github-actions bot commented May 30, 2023 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 86aaec5 Previous: 8490369 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 577.33 us/op 497.32 us/op 1.16
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 51.659 us/op 45.225 us/op 1.14
BLS verify - blst-native 1.2726 ms/op 1.1990 ms/op 1.06
BLS verifyMultipleSignatures 3 - blst-native 2.5712 ms/op 2.4431 ms/op 1.05
BLS verifyMultipleSignatures 8 - blst-native 5.4942 ms/op 5.2412 ms/op 1.05
BLS verifyMultipleSignatures 32 - blst-native 20.161 ms/op 19.026 ms/op 1.06
BLS aggregatePubkeys 32 - blst-native 26.780 us/op 25.494 us/op 1.05
BLS aggregatePubkeys 128 - blst-native 104.56 us/op 99.038 us/op 1.06
getAttestationsForBlock 61.076 ms/op 60.243 ms/op 1.01
isKnown best case - 1 super set check 275.00 ns/op 263.00 ns/op 1.05
isKnown normal case - 2 super set checks 259.00 ns/op 263.00 ns/op 0.98
isKnown worse case - 16 super set checks 257.00 ns/op 260.00 ns/op 0.99
CheckpointStateCache - add get delete 6.0070 us/op 5.7350 us/op 1.05
validate gossip signedAggregateAndProof - struct 2.9789 ms/op 2.7782 ms/op 1.07
validate gossip attestation - struct 1.4384 ms/op 1.3178 ms/op 1.09
pickEth1Vote - no votes 1.3816 ms/op 1.3828 ms/op 1.00
pickEth1Vote - max votes 9.3882 ms/op 10.783 ms/op 0.87
pickEth1Vote - Eth1Data hashTreeRoot value x2048 9.8935 ms/op 9.8935 ms/op 1.00
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 17.074 ms/op 16.273 ms/op 1.05
pickEth1Vote - Eth1Data fastSerialize value x2048 714.35 us/op 741.19 us/op 0.96
pickEth1Vote - Eth1Data fastSerialize tree x2048 9.5049 ms/op 5.7572 ms/op 1.65
bytes32 toHexString 776.00 ns/op 703.00 ns/op 1.10
bytes32 Buffer.toString(hex) 467.00 ns/op 429.00 ns/op 1.09
bytes32 Buffer.toString(hex) from Uint8Array 664.00 ns/op 613.00 ns/op 1.08
bytes32 Buffer.toString(hex) + 0x 453.00 ns/op 427.00 ns/op 1.06
Object access 1 prop 0.22400 ns/op 0.20400 ns/op 1.10
Map access 1 prop 0.17300 ns/op 0.16600 ns/op 1.04
Object get x1000 7.2070 ns/op 6.7580 ns/op 1.07
Map get x1000 0.68000 ns/op 0.61500 ns/op 1.11
Object set x1000 65.882 ns/op 68.331 ns/op 0.96
Map set x1000 49.923 ns/op 54.340 ns/op 0.92
Return object 10000 times 0.24770 ns/op 0.24640 ns/op 1.01
Throw Error 10000 times 4.3129 us/op 4.3296 us/op 1.00
fastMsgIdFn sha256 / 200 bytes 3.5760 us/op 3.5840 us/op 1.00
fastMsgIdFn h32 xxhash / 200 bytes 317.00 ns/op 328.00 ns/op 0.97
fastMsgIdFn h64 xxhash / 200 bytes 449.00 ns/op 479.00 ns/op 0.94
fastMsgIdFn sha256 / 1000 bytes 12.112 us/op 11.847 us/op 1.02
fastMsgIdFn h32 xxhash / 1000 bytes 443.00 ns/op 465.00 ns/op 0.95
fastMsgIdFn h64 xxhash / 1000 bytes 537.00 ns/op 570.00 ns/op 0.94
fastMsgIdFn sha256 / 10000 bytes 107.70 us/op 104.38 us/op 1.03
fastMsgIdFn h32 xxhash / 10000 bytes 2.0320 us/op 1.9990 us/op 1.02
fastMsgIdFn h64 xxhash / 10000 bytes 1.4520 us/op 1.4680 us/op 0.99
enrSubnets - fastDeserialize 64 bits 1.4050 us/op 1.7630 us/op 0.80
enrSubnets - ssz BitVector 64 bits 606.00 ns/op 640.00 ns/op 0.95
enrSubnets - fastDeserialize 4 bits 204.00 ns/op 215.00 ns/op 0.95
enrSubnets - ssz BitVector 4 bits 586.00 ns/op 641.00 ns/op 0.91
prioritizePeers score -10:0 att 32-0.1 sync 2-0 107.89 us/op 116.91 us/op 0.92
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 142.05 us/op 164.50 us/op 0.86
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 167.80 us/op 197.32 us/op 0.85
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 361.80 us/op 375.24 us/op 0.96
prioritizePeers score 0:0 att 64-1 sync 4-1 434.86 us/op 454.58 us/op 0.96
array of 16000 items push then shift 1.8207 us/op 1.6856 us/op 1.08
LinkedList of 16000 items push then shift 9.2800 ns/op 9.5600 ns/op 0.97
array of 16000 items push then pop 105.03 ns/op 120.49 ns/op 0.87
LinkedList of 16000 items push then pop 8.9150 ns/op 9.1090 ns/op 0.98
array of 24000 items push then shift 2.4615 us/op 2.3736 us/op 1.04
LinkedList of 24000 items push then shift 8.9760 ns/op 9.5760 ns/op 0.94
array of 24000 items push then pop 79.206 ns/op 94.468 ns/op 0.84
LinkedList of 24000 items push then pop 8.8330 ns/op 9.0590 ns/op 0.98
intersect bitArray bitLen 8 13.640 ns/op 13.504 ns/op 1.01
intersect array and set length 8 81.140 ns/op 105.85 ns/op 0.77
intersect bitArray bitLen 128 45.114 ns/op 45.217 ns/op 1.00
intersect array and set length 128 1.1262 us/op 1.3050 us/op 0.86
Buffer.concat 32 items 2.8940 us/op 3.0030 us/op 0.96
Uint8Array.set 32 items 2.7230 us/op 2.4150 us/op 1.13
transfer serialized Status (84 B) 2.0910 us/op 2.2580 us/op 0.93
copy serialized Status (84 B) 1.7650 us/op 1.9210 us/op 0.92
transfer serialized SignedVoluntaryExit (112 B) 2.1720 us/op 2.4960 us/op 0.87
copy serialized SignedVoluntaryExit (112 B) 1.8030 us/op 1.9640 us/op 0.92
transfer serialized ProposerSlashing (416 B) 2.6630 us/op 2.9260 us/op 0.91
copy serialized ProposerSlashing (416 B) 2.6830 us/op 3.2060 us/op 0.84
transfer serialized Attestation (485 B) 2.9240 us/op 3.3390 us/op 0.88
copy serialized Attestation (485 B) 2.6310 us/op 3.1090 us/op 0.85
transfer serialized AttesterSlashing (33232 B) 2.9610 us/op 2.8370 us/op 1.04
copy serialized AttesterSlashing (33232 B) 5.6900 us/op 8.9810 us/op 0.63
transfer serialized Small SignedBeaconBlock (128000 B) 3.1630 us/op 3.6780 us/op 0.86
copy serialized Small SignedBeaconBlock (128000 B) 13.942 us/op 32.235 us/op 0.43
transfer serialized Avg SignedBeaconBlock (200000 B) 3.4060 us/op 3.6390 us/op 0.94
copy serialized Avg SignedBeaconBlock (200000 B) 44.096 us/op 32.741 us/op 1.35
transfer serialized BlobsSidecar (524380 B) 3.0110 us/op 3.9790 us/op 0.76
copy serialized BlobsSidecar (524380 B) 178.97 us/op 213.59 us/op 0.84
transfer serialized Big SignedBeaconBlock (1000000 B) 3.2370 us/op 3.9490 us/op 0.82
copy serialized Big SignedBeaconBlock (1000000 B) 136.62 us/op 366.68 us/op 0.37
pass gossip attestations to forkchoice per slot 2.5627 ms/op 2.4138 ms/op 1.06
forkChoice updateHead vc 100000 bc 64 eq 0 1.9984 ms/op
forkChoice updateHead vc 600000 bc 64 eq 0 11.313 ms/op
forkChoice updateHead vc 1000000 bc 64 eq 0 21.465 ms/op
forkChoice updateHead vc 600000 bc 320 eq 0 16.180 ms/op
forkChoice updateHead vc 600000 bc 1200 eq 0 76.934 ms/op
forkChoice updateHead vc 600000 bc 64 eq 1000 19.846 ms/op
forkChoice updateHead vc 600000 bc 64 eq 10000 22.224 ms/op
forkChoice updateHead vc 600000 bc 64 eq 300000 30.908 ms/op
computeDeltas 3.7987 ms/op 3.1568 ms/op 1.20
computeProposerBoostScoreFromBalances 1.7659 ms/op 1.7702 ms/op 1.00
altair processAttestation - 250000 vs - 7PWei normalcase 2.0436 ms/op 2.3706 ms/op 0.86
altair processAttestation - 250000 vs - 7PWei worstcase 3.6169 ms/op 3.3652 ms/op 1.07
altair processAttestation - setStatus - 1/6 committees join 140.33 us/op 140.85 us/op 1.00
altair processAttestation - setStatus - 1/3 committees join 278.38 us/op 284.81 us/op 0.98
altair processAttestation - setStatus - 1/2 committees join 358.68 us/op 369.74 us/op 0.97
altair processAttestation - setStatus - 2/3 committees join 457.67 us/op 458.18 us/op 1.00
altair processAttestation - setStatus - 4/5 committees join 634.01 us/op 661.12 us/op 0.96
altair processAttestation - setStatus - 100% committees join 771.71 us/op 745.14 us/op 1.04
altair processBlock - 250000 vs - 7PWei normalcase 17.711 ms/op 14.458 ms/op 1.22
altair processBlock - 250000 vs - 7PWei normalcase hashState 27.945 ms/op 25.233 ms/op 1.11
altair processBlock - 250000 vs - 7PWei worstcase 46.746 ms/op 46.349 ms/op 1.01
altair processBlock - 250000 vs - 7PWei worstcase hashState 65.705 ms/op 68.550 ms/op 0.96
phase0 processBlock - 250000 vs - 7PWei normalcase 2.2598 ms/op 2.3659 ms/op 0.96
phase0 processBlock - 250000 vs - 7PWei worstcase 29.779 ms/op 29.189 ms/op 1.02
altair processEth1Data - 250000 vs - 7PWei normalcase 473.23 us/op 487.47 us/op 0.97
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 7.0090 us/op 7.7250 us/op 0.91
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 20.020 us/op 22.694 us/op 0.88
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 9.0180 us/op 9.0060 us/op 1.00
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 6.4570 us/op 6.9600 us/op 0.93
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 74.636 us/op 90.176 us/op 0.83
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 601.94 us/op 626.61 us/op 0.96
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 871.94 us/op 896.49 us/op 0.97
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 890.33 us/op 868.37 us/op 1.03
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 2.1150 ms/op 2.1902 ms/op 0.97
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 1.5354 ms/op 1.4941 ms/op 1.03
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 3.7618 ms/op 4.1218 ms/op 0.91
Tree 40 250000 create 297.85 ms/op 309.05 ms/op 0.96
Tree 40 250000 get(125000) 182.21 ns/op 176.44 ns/op 1.03
Tree 40 250000 set(125000) 917.39 ns/op 892.00 ns/op 1.03
Tree 40 250000 toArray() 17.506 ms/op 17.256 ms/op 1.01
Tree 40 250000 iterate all - toArray() + loop 17.692 ms/op 17.051 ms/op 1.04
Tree 40 250000 iterate all - get(i) 67.487 ms/op 65.847 ms/op 1.02
MutableVector 250000 create 10.861 ms/op 10.563 ms/op 1.03
MutableVector 250000 get(125000) 6.3610 ns/op 6.2540 ns/op 1.02
MutableVector 250000 set(125000) 250.58 ns/op 278.05 ns/op 0.90
MutableVector 250000 toArray() 2.6920 ms/op 3.0332 ms/op 0.89
MutableVector 250000 iterate all - toArray() + loop 2.7134 ms/op 2.8428 ms/op 0.95
MutableVector 250000 iterate all - get(i) 1.5302 ms/op 1.5162 ms/op 1.01
Array 250000 create 2.5160 ms/op 2.5604 ms/op 0.98
Array 250000 clone - spread 1.2375 ms/op 1.1122 ms/op 1.11
Array 250000 get(125000) 0.59900 ns/op 0.56100 ns/op 1.07
Array 250000 set(125000) 0.66900 ns/op 0.63000 ns/op 1.06
Array 250000 iterate all - loop 79.511 us/op 110.25 us/op 0.72
effectiveBalanceIncrements clone Uint8Array 300000 33.737 us/op 27.056 us/op 1.25
effectiveBalanceIncrements clone MutableVector 300000 392.00 ns/op 345.00 ns/op 1.14
effectiveBalanceIncrements rw all Uint8Array 300000 161.69 us/op 168.05 us/op 0.96
effectiveBalanceIncrements rw all MutableVector 300000 80.996 ms/op 81.407 ms/op 0.99
phase0 afterProcessEpoch - 250000 vs - 7PWei 114.61 ms/op 114.35 ms/op 1.00
phase0 beforeProcessEpoch - 250000 vs - 7PWei 36.355 ms/op 41.838 ms/op 0.87
altair processEpoch - mainnet_e81889 332.65 ms/op 305.75 ms/op 1.09
mainnet_e81889 - altair beforeProcessEpoch 64.164 ms/op 69.173 ms/op 0.93
mainnet_e81889 - altair processJustificationAndFinalization 19.484 us/op 18.213 us/op 1.07
mainnet_e81889 - altair processInactivityUpdates 5.6187 ms/op 6.3750 ms/op 0.88
mainnet_e81889 - altair processRewardsAndPenalties 49.648 ms/op 52.945 ms/op 0.94
mainnet_e81889 - altair processRegistryUpdates 2.7880 us/op 3.6140 us/op 0.77
mainnet_e81889 - altair processSlashings 432.00 ns/op 498.00 ns/op 0.87
mainnet_e81889 - altair processEth1DataReset 514.00 ns/op 548.00 ns/op 0.94
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.2541 ms/op 1.2494 ms/op 1.00
mainnet_e81889 - altair processSlashingsReset 4.9650 us/op 4.7570 us/op 1.04
mainnet_e81889 - altair processRandaoMixesReset 4.6340 us/op 4.5270 us/op 1.02
mainnet_e81889 - altair processHistoricalRootsUpdate 589.00 ns/op 613.00 ns/op 0.96
mainnet_e81889 - altair processParticipationFlagUpdates 2.7220 us/op 2.4380 us/op 1.12
mainnet_e81889 - altair processSyncCommitteeUpdates 1.0700 us/op 880.00 ns/op 1.22
mainnet_e81889 - altair afterProcessEpoch 134.47 ms/op 126.56 ms/op 1.06
phase0 processEpoch - mainnet_e58758 375.53 ms/op 357.97 ms/op 1.05
mainnet_e58758 - phase0 beforeProcessEpoch 148.59 ms/op 140.42 ms/op 1.06
mainnet_e58758 - phase0 processJustificationAndFinalization 18.201 us/op 17.104 us/op 1.06
mainnet_e58758 - phase0 processRewardsAndPenalties 64.423 ms/op 65.821 ms/op 0.98
mainnet_e58758 - phase0 processRegistryUpdates 8.8770 us/op 8.0200 us/op 1.11
mainnet_e58758 - phase0 processSlashings 517.00 ns/op 489.00 ns/op 1.06
mainnet_e58758 - phase0 processEth1DataReset 569.00 ns/op 494.00 ns/op 1.15
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.0202 ms/op 1.0617 ms/op 0.96
mainnet_e58758 - phase0 processSlashingsReset 4.0690 us/op 3.3670 us/op 1.21
mainnet_e58758 - phase0 processRandaoMixesReset 4.3880 us/op 4.4460 us/op 0.99
mainnet_e58758 - phase0 processHistoricalRootsUpdate 587.00 ns/op 588.00 ns/op 1.00
mainnet_e58758 - phase0 processParticipationRecordUpdates 4.1800 us/op 3.8130 us/op 1.10
mainnet_e58758 - phase0 afterProcessEpoch 102.34 ms/op 97.858 ms/op 1.05
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.2774 ms/op 1.2303 ms/op 1.04
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.4648 ms/op 1.5625 ms/op 0.94
altair processInactivityUpdates - 250000 normalcase 26.237 ms/op 28.439 ms/op 0.92
altair processInactivityUpdates - 250000 worstcase 26.259 ms/op 29.266 ms/op 0.90
phase0 processRegistryUpdates - 250000 normalcase 7.8170 us/op 6.8990 us/op 1.13
phase0 processRegistryUpdates - 250000 badcase_full_deposits 269.01 us/op 268.72 us/op 1.00
phase0 processRegistryUpdates - 250000 worstcase 0.5 132.36 ms/op 130.13 ms/op 1.02
altair processRewardsAndPenalties - 250000 normalcase 63.489 ms/op 68.495 ms/op 0.93
altair processRewardsAndPenalties - 250000 worstcase 70.798 ms/op 72.642 ms/op 0.97
phase0 getAttestationDeltas - 250000 normalcase 6.7134 ms/op 6.8962 ms/op 0.97
phase0 getAttestationDeltas - 250000 worstcase 6.7796 ms/op 6.8561 ms/op 0.99
phase0 processSlashings - 250000 worstcase 3.8536 ms/op 3.5053 ms/op 1.10
altair processSyncCommitteeUpdates - 250000 182.93 ms/op 180.48 ms/op 1.01
BeaconState.hashTreeRoot - No change 267.00 ns/op 261.00 ns/op 1.02
BeaconState.hashTreeRoot - 1 full validator 54.945 us/op 52.156 us/op 1.05
BeaconState.hashTreeRoot - 32 full validator 493.06 us/op 474.31 us/op 1.04
BeaconState.hashTreeRoot - 512 full validator 5.5564 ms/op 5.7647 ms/op 0.96
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 63.944 us/op 61.876 us/op 1.03
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 908.71 us/op 899.15 us/op 1.01
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 11.595 ms/op 11.172 ms/op 1.04
BeaconState.hashTreeRoot - 1 balances 48.986 us/op 50.700 us/op 0.97
BeaconState.hashTreeRoot - 32 balances 451.63 us/op 463.54 us/op 0.97
BeaconState.hashTreeRoot - 512 balances 4.4046 ms/op 4.1975 ms/op 1.05
BeaconState.hashTreeRoot - 250000 balances 74.072 ms/op 70.520 ms/op 1.05
aggregationBits - 2048 els - zipIndexesInBitList 16.427 us/op 16.839 us/op 0.98
regular array get 100000 times 43.668 us/op 32.601 us/op 1.34
wrappedArray get 100000 times 33.113 us/op 32.552 us/op 1.02
arrayWithProxy get 100000 times 15.337 ms/op 16.305 ms/op 0.94
ssz.Root.equals 570.00 ns/op 569.00 ns/op 1.00
byteArrayEquals 526.00 ns/op 546.00 ns/op 0.96
shuffle list - 16384 els 6.8503 ms/op 7.0448 ms/op 0.97
shuffle list - 250000 els 100.71 ms/op 99.512 ms/op 1.01
processSlot - 1 slots 8.5900 us/op 8.8070 us/op 0.98
processSlot - 32 slots 1.3053 ms/op 1.3720 ms/op 0.95
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 35.529 ms/op 36.527 ms/op 0.97
getCommitteeAssignments - req 1 vs - 250000 vc 2.9844 ms/op 2.8883 ms/op 1.03
getCommitteeAssignments - req 100 vs - 250000 vc 4.2158 ms/op 4.0972 ms/op 1.03
getCommitteeAssignments - req 1000 vs - 250000 vc 4.5639 ms/op 4.3645 ms/op 1.05
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.2000 ns/op 4.7900 ns/op 1.09
state getBlockRootAtSlot - 250000 vs - 7PWei 1.0681 us/op 797.70 ns/op 1.34
computeProposers - vc 250000 10.985 ms/op 10.567 ms/op 1.04
computeEpochShuffling - vc 250000 105.77 ms/op 103.35 ms/op 1.02
getNextSyncCommittee - vc 250000 173.88 ms/op 172.99 ms/op 1.01
computeSigningRoot for AttestationData 13.467 us/op 12.757 us/op 1.06
hash AttestationData serialized data then Buffer.toString(base64) 2.4980 us/op 2.4535 us/op 1.02
toHexString serialized data 1.0652 us/op 1.0696 us/op 1.00
Buffer.toString(base64) 328.29 ns/op 319.15 ns/op 1.03

by benchmarkbot/action

@dapplion dapplion mentioned this pull request May 30, 2023
Closed
@wemeetagain wemeetagain changed the title Add fork-choice benchmark for updateHead test: add fork-choice benchmark for updateHead May 31, 2023
@wemeetagain wemeetagain merged commit b7bedf1 into unstable May 31, 2023
@wemeetagain wemeetagain deleted the dapplion/fork-choice-benchmarks branch May 31, 2023 18:51
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.9.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@dapplion @wemeetagain