Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

test: consolidate sim-merge tests #6344

Merged
merged 7 commits into from
Feb 26, 2024
Merged

test: consolidate sim-merge tests #6344

merged 7 commits into from
Feb 26, 2024

Conversation

nazarhussain
Copy link
Contributor

Motivation

Consolidate the sim tests, to minimize the CI run time and reduce maintenance effort.

Description

  • Move assertions from merge-interop.test.ts to multi_fork.test.ts
  • Remove merge-interop.test.ts

Steps to test or reproduce

  • Run all tests

@nazarhussain nazarhussain requested a review from a team as a code owner January 23, 2024 15:24
@nazarhussain nazarhussain self-assigned this Jan 23, 2024
@nazarhussain
Copy link
Contributor Author

@g11tech Please review, if there is any assertion left in merge-interop.test.ts but not covered in multi_fork.test.ts

@codecov Codecov
Copy link

codecov bot commented Jan 23, 2024 •
edited
Loading

Codecov Report

Merging #6344 (43eeb35) into unstable (8959bda) will not change coverage.
The diff coverage is n/a.

Additional details and impacted files 
@@            Coverage Diff            @@
##           unstable    #6344   +/-   ##
=========================================
  Coverage     61.66%   61.66%           
=========================================
  Files           554      554           
  Lines         58078    58078           
  Branches       1837     1837           
=========================================
  Hits          35811    35811           
  Misses        22228    22228           
  Partials         39       39

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 23, 2024 •
edited
Loading

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: e1e80a3 Previous: 8959bda Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 511.84 us/op 386.77 us/op 1.32
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 73.156 us/op 52.713 us/op 1.39
BLS verify - blst-native 1.1548 ms/op 1.1065 ms/op 1.04
BLS verifyMultipleSignatures 3 - blst-native 2.4538 ms/op 2.4125 ms/op 1.02
BLS verifyMultipleSignatures 8 - blst-native 5.6362 ms/op 5.2622 ms/op 1.07
BLS verifyMultipleSignatures 32 - blst-native 20.571 ms/op 19.393 ms/op 1.06
BLS verifyMultipleSignatures 64 - blst-native 39.669 ms/op 38.213 ms/op 1.04
BLS verifyMultipleSignatures 128 - blst-native 81.831 ms/op 75.454 ms/op 1.08
BLS deserializing 10000 signatures 837.82 ms/op 797.14 ms/op 1.05
BLS deserializing 100000 signatures 8.5013 s/op 8.2573 s/op 1.03
BLS verifyMultipleSignatures - same message - 3 - blst-native 1.1459 ms/op 1.1121 ms/op 1.03
BLS verifyMultipleSignatures - same message - 8 - blst-native 1.2897 ms/op 1.2572 ms/op 1.03
BLS verifyMultipleSignatures - same message - 32 - blst-native 2.4471 ms/op 1.9712 ms/op 1.24
BLS verifyMultipleSignatures - same message - 64 - blst-native 3.6325 ms/op 3.7683 ms/op 0.96
BLS verifyMultipleSignatures - same message - 128 - blst-native 6.8379 ms/op 4.8938 ms/op 1.40
BLS aggregatePubkeys 32 - blst-native 24.015 us/op 22.071 us/op 1.09
BLS aggregatePubkeys 128 - blst-native 88.060 us/op 86.402 us/op 1.02
notSeenSlots=1 numMissedVotes=1 numBadVotes=10 43.769 ms/op 45.012 ms/op 0.97
notSeenSlots=1 numMissedVotes=0 numBadVotes=4 43.825 ms/op 42.561 ms/op 1.03
notSeenSlots=2 numMissedVotes=1 numBadVotes=10 26.154 ms/op 25.926 ms/op 1.01
getSlashingsAndExits - default max 105.58 us/op 223.06 us/op 0.47
getSlashingsAndExits - 2k 338.97 us/op 350.91 us/op 0.97
proposeBlockBody type=full, size=empty 4.0674 ms/op 3.7844 ms/op 1.07
isKnown best case - 1 super set check 319.00 ns/op 474.00 ns/op 0.67
isKnown normal case - 2 super set checks 316.00 ns/op 389.00 ns/op 0.81
isKnown worse case - 16 super set checks 336.00 ns/op 340.00 ns/op 0.99
CheckpointStateCache - add get delete 4.2240 us/op 3.7860 us/op 1.12
validate api signedAggregateAndProof - struct 2.4373 ms/op 2.3719 ms/op 1.03
validate gossip signedAggregateAndProof - struct 2.5739 ms/op 2.2532 ms/op 1.14
validate gossip attestation - vc 640000 1.1787 ms/op 1.1302 ms/op 1.04
batch validate gossip attestation - vc 640000 - chunk 32 144.19 us/op 138.19 us/op 1.04
batch validate gossip attestation - vc 640000 - chunk 64 125.55 us/op 128.71 us/op 0.98
batch validate gossip attestation - vc 640000 - chunk 128 114.32 us/op 124.56 us/op 0.92
batch validate gossip attestation - vc 640000 - chunk 256 109.88 us/op 114.03 us/op 0.96
pickEth1Vote - no votes 855.38 us/op 874.46 us/op 0.98
pickEth1Vote - max votes 9.2448 ms/op 9.2615 ms/op 1.00
pickEth1Vote - Eth1Data hashTreeRoot value x2048 16.311 ms/op 21.090 ms/op 0.77
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 18.008 ms/op 26.025 ms/op 0.69
pickEth1Vote - Eth1Data fastSerialize value x2048 447.52 us/op 412.58 us/op 1.08
pickEth1Vote - Eth1Data fastSerialize tree x2048 3.3379 ms/op 5.8079 ms/op 0.57
bytes32 toHexString 453.00 ns/op 458.00 ns/op 0.99
bytes32 Buffer.toString(hex) 321.00 ns/op 311.00 ns/op 1.03
bytes32 Buffer.toString(hex) from Uint8Array 520.00 ns/op 447.00 ns/op 1.16
bytes32 Buffer.toString(hex) + 0x 323.00 ns/op 309.00 ns/op 1.05
Object access 1 prop 0.20800 ns/op 0.22000 ns/op 0.95
Map access 1 prop 0.19800 ns/op 0.20700 ns/op 0.96
Object get x1000 5.4830 ns/op 4.8920 ns/op 1.12
Map get x1000 0.73100 ns/op 0.75400 ns/op 0.97
Object set x1000 27.850 ns/op 25.506 ns/op 1.09
Map set x1000 16.967 ns/op 18.433 ns/op 0.92
Return object 10000 times 0.22860 ns/op 0.22100 ns/op 1.03
Throw Error 10000 times 2.7212 us/op 2.6150 us/op 1.04
fastMsgIdFn sha256 / 200 bytes 1.9300 us/op 1.9720 us/op 0.98
fastMsgIdFn h32 xxhash / 200 bytes 286.00 ns/op 322.00 ns/op 0.89
fastMsgIdFn h64 xxhash / 200 bytes 343.00 ns/op 340.00 ns/op 1.01
fastMsgIdFn sha256 / 1000 bytes 6.0620 us/op 5.9550 us/op 1.02
fastMsgIdFn h32 xxhash / 1000 bytes 400.00 ns/op 446.00 ns/op 0.90
fastMsgIdFn h64 xxhash / 1000 bytes 432.00 ns/op 427.00 ns/op 1.01
fastMsgIdFn sha256 / 10000 bytes 52.587 us/op 50.483 us/op 1.04
fastMsgIdFn h32 xxhash / 10000 bytes 1.9010 us/op 1.7950 us/op 1.06
fastMsgIdFn h64 xxhash / 10000 bytes 1.2940 us/op 1.2490 us/op 1.04
send data - 1000 256B messages 11.163 ms/op 13.509 ms/op 0.83
send data - 1000 512B messages 14.664 ms/op 16.686 ms/op 0.88
send data - 1000 1024B messages 25.157 ms/op 31.195 ms/op 0.81
send data - 1000 1200B messages 31.527 ms/op 30.750 ms/op 1.03
send data - 1000 2048B messages 42.196 ms/op 38.646 ms/op 1.09
send data - 1000 4096B messages 39.932 ms/op 38.966 ms/op 1.02
send data - 1000 16384B messages 95.218 ms/op 92.689 ms/op 1.03
send data - 1000 65536B messages 384.78 ms/op 456.11 ms/op 0.84
enrSubnets - fastDeserialize 64 bits 1.4040 us/op 1.0360 us/op 1.36
enrSubnets - ssz BitVector 64 bits 565.00 ns/op 502.00 ns/op 1.13
enrSubnets - fastDeserialize 4 bits 274.00 ns/op 254.00 ns/op 1.08
enrSubnets - ssz BitVector 4 bits 566.00 ns/op 515.00 ns/op 1.10
prioritizePeers score -10:0 att 32-0.1 sync 2-0 77.796 us/op 103.95 us/op 0.75
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 95.967 us/op 114.04 us/op 0.84
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 145.23 us/op 146.76 us/op 0.99
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 225.88 us/op 260.80 us/op 0.87
prioritizePeers score 0:0 att 64-1 sync 4-1 218.05 us/op 277.13 us/op 0.79
array of 16000 items push then shift 1.3249 us/op 1.3783 us/op 0.96
LinkedList of 16000 items push then shift 6.0530 ns/op 8.5800 ns/op 0.71
array of 16000 items push then pop 83.422 ns/op 113.41 ns/op 0.74
LinkedList of 16000 items push then pop 5.8610 ns/op 7.7260 ns/op 0.76
array of 24000 items push then shift 2.0754 us/op 1.8896 us/op 1.10
LinkedList of 24000 items push then shift 7.8300 ns/op 7.5310 ns/op 1.04
array of 24000 items push then pop 137.47 ns/op 150.20 ns/op 0.92
LinkedList of 24000 items push then pop 5.8730 ns/op 6.0850 ns/op 0.97
intersect bitArray bitLen 8 5.1420 ns/op 4.9080 ns/op 1.05
intersect array and set length 8 49.872 ns/op 50.458 ns/op 0.99
intersect bitArray bitLen 128 29.597 ns/op 29.444 ns/op 1.01
intersect array and set length 128 717.78 ns/op 696.32 ns/op 1.03
bitArray.getTrueBitIndexes() bitLen 128 1.5730 us/op 1.3890 us/op 1.13
bitArray.getTrueBitIndexes() bitLen 248 2.6380 us/op 2.9850 us/op 0.88
bitArray.getTrueBitIndexes() bitLen 512 5.6850 us/op 6.3790 us/op 0.89
Buffer.concat 32 items 1.0120 us/op 1.0400 us/op 0.97
Uint8Array.set 32 items 1.5830 us/op 2.3270 us/op 0.68
Set add up to 64 items then delete first 1.9689 us/op 2.4738 us/op 0.80
OrderedSet add up to 64 items then delete first 2.8300 us/op 3.8685 us/op 0.73
Set add up to 64 items then delete last 2.0390 us/op 2.7309 us/op 0.75
OrderedSet add up to 64 items then delete last 3.0192 us/op 3.7934 us/op 0.80
Set add up to 64 items then delete middle 1.9949 us/op 2.3774 us/op 0.84
OrderedSet add up to 64 items then delete middle 4.2821 us/op 5.2910 us/op 0.81
Set add up to 128 items then delete first 3.9680 us/op 5.0557 us/op 0.78
OrderedSet add up to 128 items then delete first 6.4682 us/op 8.5599 us/op 0.76
Set add up to 128 items then delete last 3.9819 us/op 5.4318 us/op 0.73
OrderedSet add up to 128 items then delete last 5.8735 us/op 8.5114 us/op 0.69
Set add up to 128 items then delete middle 3.9452 us/op 5.1232 us/op 0.77
OrderedSet add up to 128 items then delete middle 11.087 us/op 13.268 us/op 0.84
Set add up to 256 items then delete first 7.8299 us/op 11.025 us/op 0.71
OrderedSet add up to 256 items then delete first 12.614 us/op 16.499 us/op 0.76
Set add up to 256 items then delete last 9.9943 us/op 10.382 us/op 0.96
OrderedSet add up to 256 items then delete last 15.541 us/op 15.686 us/op 0.99
Set add up to 256 items then delete middle 8.8230 us/op 10.915 us/op 0.81
OrderedSet add up to 256 items then delete middle 38.468 us/op 36.435 us/op 1.06
transfer serialized Status (84 B) 1.5140 us/op 1.5920 us/op 0.95
copy serialized Status (84 B) 1.2440 us/op 1.3100 us/op 0.95
transfer serialized SignedVoluntaryExit (112 B) 1.6660 us/op 1.7400 us/op 0.96
copy serialized SignedVoluntaryExit (112 B) 1.4560 us/op 1.3480 us/op 1.08
transfer serialized ProposerSlashing (416 B) 1.8860 us/op 2.3590 us/op 0.80
copy serialized ProposerSlashing (416 B) 2.3930 us/op 2.4810 us/op 0.96
transfer serialized Attestation (485 B) 2.6230 us/op 2.7570 us/op 0.95
copy serialized Attestation (485 B) 2.4100 us/op 2.6620 us/op 0.91
transfer serialized AttesterSlashing (33232 B) 2.3630 us/op 2.8950 us/op 0.82
copy serialized AttesterSlashing (33232 B) 4.9060 us/op 10.299 us/op 0.48
transfer serialized Small SignedBeaconBlock (128000 B) 2.8630 us/op 2.8030 us/op 1.02
copy serialized Small SignedBeaconBlock (128000 B) 9.6000 us/op 11.933 us/op 0.80
transfer serialized Avg SignedBeaconBlock (200000 B) 2.8280 us/op 2.4200 us/op 1.17
copy serialized Avg SignedBeaconBlock (200000 B) 28.627 us/op 56.162 us/op 0.51
transfer serialized BlobsSidecar (524380 B) 2.4470 us/op 2.6150 us/op 0.94
copy serialized BlobsSidecar (524380 B) 89.929 us/op 75.719 us/op 1.19
transfer serialized Big SignedBeaconBlock (1000000 B) 2.9240 us/op 2.6830 us/op 1.09
copy serialized Big SignedBeaconBlock (1000000 B) 139.73 us/op 292.08 us/op 0.48
pass gossip attestations to forkchoice per slot 2.7956 ms/op 2.9304 ms/op 0.95
forkChoice updateHead vc 100000 bc 64 eq 0 462.60 us/op 492.15 us/op 0.94
forkChoice updateHead vc 600000 bc 64 eq 0 2.8154 ms/op 3.2767 ms/op 0.86
forkChoice updateHead vc 1000000 bc 64 eq 0 4.6151 ms/op 6.3741 ms/op 0.72
forkChoice updateHead vc 600000 bc 320 eq 0 2.7194 ms/op 2.6594 ms/op 1.02
forkChoice updateHead vc 600000 bc 1200 eq 0 2.7932 ms/op 2.8068 ms/op 1.00
forkChoice updateHead vc 600000 bc 7200 eq 0 3.4397 ms/op 3.5648 ms/op 0.96
forkChoice updateHead vc 600000 bc 64 eq 1000 9.7448 ms/op 9.8837 ms/op 0.99
forkChoice updateHead vc 600000 bc 64 eq 10000 9.6893 ms/op 9.6663 ms/op 1.00
forkChoice updateHead vc 600000 bc 64 eq 300000 12.172 ms/op 12.158 ms/op 1.00
computeDeltas 500000 validators 300 proto nodes 3.1760 ms/op 3.3531 ms/op 0.95
computeDeltas 500000 validators 1200 proto nodes 3.2363 ms/op 3.3802 ms/op 0.96
computeDeltas 500000 validators 7200 proto nodes 3.3488 ms/op 3.2492 ms/op 1.03
computeDeltas 750000 validators 300 proto nodes 5.1104 ms/op 5.4233 ms/op 0.94
computeDeltas 750000 validators 1200 proto nodes 4.8127 ms/op 5.1672 ms/op 0.93
computeDeltas 750000 validators 7200 proto nodes 4.9322 ms/op 5.5880 ms/op 0.88
computeDeltas 1400000 validators 300 proto nodes 9.8280 ms/op 10.850 ms/op 0.91
computeDeltas 1400000 validators 1200 proto nodes 11.767 ms/op 10.587 ms/op 1.11
computeDeltas 1400000 validators 7200 proto nodes 12.357 ms/op 9.3930 ms/op 1.32
computeDeltas 2100000 validators 300 proto nodes 16.086 ms/op 15.580 ms/op 1.03
computeDeltas 2100000 validators 1200 proto nodes 15.142 ms/op 16.707 ms/op 0.91
computeDeltas 2100000 validators 7200 proto nodes 14.291 ms/op 14.448 ms/op 0.99
altair processAttestation - 250000 vs - 7PWei normalcase 2.0726 ms/op 1.5358 ms/op 1.35
altair processAttestation - 250000 vs - 7PWei worstcase 3.0371 ms/op 2.6276 ms/op 1.16
altair processAttestation - setStatus - 1/6 committees join 107.45 us/op 75.732 us/op 1.42
altair processAttestation - setStatus - 1/3 committees join 203.89 us/op 158.53 us/op 1.29
altair processAttestation - setStatus - 1/2 committees join 290.20 us/op 214.24 us/op 1.35
altair processAttestation - setStatus - 2/3 committees join 378.86 us/op 284.28 us/op 1.33
altair processAttestation - setStatus - 4/5 committees join 514.32 us/op 446.22 us/op 1.15
altair processAttestation - setStatus - 100% committees join 614.40 us/op 555.52 us/op 1.11
altair processBlock - 250000 vs - 7PWei normalcase 12.771 ms/op 8.8400 ms/op 1.44
altair processBlock - 250000 vs - 7PWei normalcase hashState 43.261 ms/op 23.593 ms/op 1.83
altair processBlock - 250000 vs - 7PWei worstcase 41.896 ms/op 28.896 ms/op 1.45
altair processBlock - 250000 vs - 7PWei worstcase hashState 94.460 ms/op 73.332 ms/op 1.29
phase0 processBlock - 250000 vs - 7PWei normalcase 3.1917 ms/op 2.5752 ms/op 1.24
phase0 processBlock - 250000 vs - 7PWei worstcase 31.678 ms/op 24.642 ms/op 1.29
altair processEth1Data - 250000 vs - 7PWei normalcase 323.69 us/op 304.15 us/op 1.06
getExpectedWithdrawals 250000 eb:1,eth1:1,we:0,wn:0,smpl:15 9.2120 us/op 14.982 us/op 0.61
getExpectedWithdrawals 250000 eb:0.95,eth1:0.1,we:0.05,wn:0,smpl:219 55.514 us/op 57.881 us/op 0.96
getExpectedWithdrawals 250000 eb:0.95,eth1:0.3,we:0.05,wn:0,smpl:42 16.280 us/op 21.257 us/op 0.77
getExpectedWithdrawals 250000 eb:0.95,eth1:0.7,we:0.05,wn:0,smpl:18 13.350 us/op 8.3040 us/op 1.61
getExpectedWithdrawals 250000 eb:0.1,eth1:0.1,we:0,wn:0,smpl:1020 255.89 us/op 155.26 us/op 1.65
getExpectedWithdrawals 250000 eb:0.03,eth1:0.03,we:0,wn:0,smpl:11777 1.4270 ms/op 1.0416 ms/op 1.37
getExpectedWithdrawals 250000 eb:0.01,eth1:0.01,we:0,wn:0,smpl:16384 1.8450 ms/op 1.4487 ms/op 1.27
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,smpl:16384 1.2689 ms/op 1.0319 ms/op 1.23
getExpectedWithdrawals 250000 eb:0,eth1:0,we:0,wn:0,nocache,smpl:16384 4.2699 ms/op 3.0478 ms/op 1.40
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,smpl:16384 3.0808 ms/op 1.5104 ms/op 2.04
getExpectedWithdrawals 250000 eb:0,eth1:1,we:0,wn:0,nocache,smpl:16384 11.064 ms/op 5.4104 ms/op 2.04
Tree 40 250000 create 519.39 ms/op 348.45 ms/op 1.49
Tree 40 250000 get(125000) 131.94 ns/op 122.11 ns/op 1.08
Tree 40 250000 set(125000) 957.72 ns/op 841.70 ns/op 1.14
Tree 40 250000 toArray() 21.592 ms/op 25.457 ms/op 0.85
Tree 40 250000 iterate all - toArray() + loop 25.521 ms/op 26.073 ms/op 0.98
Tree 40 250000 iterate all - get(i) 64.280 ms/op 54.605 ms/op 1.18
MutableVector 250000 create 11.518 ms/op 12.246 ms/op 0.94
MutableVector 250000 get(125000) 6.0920 ns/op 5.7150 ns/op 1.07
MutableVector 250000 set(125000) 478.38 ns/op 221.35 ns/op 2.16
MutableVector 250000 toArray() 3.0500 ms/op 2.4479 ms/op 1.25
MutableVector 250000 iterate all - toArray() + loop 2.9505 ms/op 2.4492 ms/op 1.20
MutableVector 250000 iterate all - get(i) 1.4153 ms/op 1.3476 ms/op 1.05
Array 250000 create 3.8262 ms/op 2.3086 ms/op 1.66
Array 250000 clone - spread 1.2644 ms/op 1.1861 ms/op 1.07
Array 250000 get(125000) 1.0600 ns/op 1.0660 ns/op 0.99
Array 250000 set(125000) 1.3390 ns/op 2.0970 ns/op 0.64
Array 250000 iterate all - loop 160.54 us/op 155.19 us/op 1.03
effectiveBalanceIncrements clone Uint8Array 300000 13.884 us/op 53.953 us/op 0.26
effectiveBalanceIncrements clone MutableVector 300000 439.00 ns/op 906.00 ns/op 0.48
effectiveBalanceIncrements rw all Uint8Array 300000 197.52 us/op 184.71 us/op 1.07
effectiveBalanceIncrements rw all MutableVector 300000 69.791 ms/op 113.57 ms/op 0.61
phase0 afterProcessEpoch - 250000 vs - 7PWei 80.963 ms/op 78.605 ms/op 1.03
phase0 beforeProcessEpoch - 250000 vs - 7PWei 62.411 ms/op 35.742 ms/op 1.75
altair processEpoch - mainnet_e81889 380.72 ms/op 472.21 ms/op 0.81
mainnet_e81889 - altair beforeProcessEpoch 65.603 ms/op 84.059 ms/op 0.78
mainnet_e81889 - altair processJustificationAndFinalization 11.961 us/op 14.334 us/op 0.83
mainnet_e81889 - altair processInactivityUpdates 4.3365 ms/op 3.8011 ms/op 1.14
mainnet_e81889 - altair processRewardsAndPenalties 49.779 ms/op 54.907 ms/op 0.91
mainnet_e81889 - altair processRegistryUpdates 2.0020 us/op 2.1410 us/op 0.94
mainnet_e81889 - altair processSlashings 328.00 ns/op 641.00 ns/op 0.51
mainnet_e81889 - altair processEth1DataReset 630.00 ns/op 720.00 ns/op 0.88
mainnet_e81889 - altair processEffectiveBalanceUpdates 1.0509 ms/op 1.0622 ms/op 0.99
mainnet_e81889 - altair processSlashingsReset 3.8740 us/op 2.8640 us/op 1.35
mainnet_e81889 - altair processRandaoMixesReset 6.3180 us/op 4.4560 us/op 1.42
mainnet_e81889 - altair processHistoricalRootsUpdate 1.0440 us/op 1.0610 us/op 0.98
mainnet_e81889 - altair processParticipationFlagUpdates 2.4800 us/op 2.0930 us/op 1.18
mainnet_e81889 - altair processSyncCommitteeUpdates 941.00 ns/op 800.00 ns/op 1.18
mainnet_e81889 - altair afterProcessEpoch 83.531 ms/op 80.468 ms/op 1.04
capella processEpoch - mainnet_e217614 2.3950 s/op 2.2908 s/op 1.05
mainnet_e217614 - capella beforeProcessEpoch 586.21 ms/op 493.64 ms/op 1.19
mainnet_e217614 - capella processJustificationAndFinalization 17.593 us/op 11.707 us/op 1.50
mainnet_e217614 - capella processInactivityUpdates 20.575 ms/op 16.809 ms/op 1.22
mainnet_e217614 - capella processRewardsAndPenalties 377.89 ms/op 408.24 ms/op 0.93
mainnet_e217614 - capella processRegistryUpdates 21.009 us/op 26.900 us/op 0.78
mainnet_e217614 - capella processSlashings 597.00 ns/op 834.00 ns/op 0.72
mainnet_e217614 - capella processEth1DataReset 718.00 ns/op 607.00 ns/op 1.18
mainnet_e217614 - capella processEffectiveBalanceUpdates 4.8152 ms/op 3.5460 ms/op 1.36
mainnet_e217614 - capella processSlashingsReset 4.4420 us/op 4.0820 us/op 1.09
mainnet_e217614 - capella processRandaoMixesReset 4.4650 us/op 6.3540 us/op 0.70
mainnet_e217614 - capella processHistoricalRootsUpdate 866.00 ns/op 740.00 ns/op 1.17
mainnet_e217614 - capella processParticipationFlagUpdates 1.9550 us/op 3.4750 us/op 0.56
mainnet_e217614 - capella afterProcessEpoch 223.11 ms/op 249.28 ms/op 0.90
phase0 processEpoch - mainnet_e58758 480.79 ms/op 481.82 ms/op 1.00
mainnet_e58758 - phase0 beforeProcessEpoch 148.81 ms/op 155.49 ms/op 0.96
mainnet_e58758 - phase0 processJustificationAndFinalization 21.724 us/op 17.263 us/op 1.26
mainnet_e58758 - phase0 processRewardsAndPenalties 58.821 ms/op 48.699 ms/op 1.21
mainnet_e58758 - phase0 processRegistryUpdates 13.494 us/op 10.676 us/op 1.26
mainnet_e58758 - phase0 processSlashings 860.00 ns/op 650.00 ns/op 1.32
mainnet_e58758 - phase0 processEth1DataReset 852.00 ns/op 918.00 ns/op 0.93
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.1551 ms/op 1.2024 ms/op 0.96
mainnet_e58758 - phase0 processSlashingsReset 4.7120 us/op 3.3100 us/op 1.42
mainnet_e58758 - phase0 processRandaoMixesReset 6.1810 us/op 4.5170 us/op 1.37
mainnet_e58758 - phase0 processHistoricalRootsUpdate 914.00 ns/op 808.00 ns/op 1.13
mainnet_e58758 - phase0 processParticipationRecordUpdates 7.6900 us/op 5.0270 us/op 1.53
mainnet_e58758 - phase0 afterProcessEpoch 66.036 ms/op 69.275 ms/op 0.95
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.4847 ms/op 1.1321 ms/op 1.31
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 1.1127 ms/op 1.1581 ms/op 0.96
altair processInactivityUpdates - 250000 normalcase 25.287 ms/op 24.925 ms/op 1.01
altair processInactivityUpdates - 250000 worstcase 30.525 ms/op 25.949 ms/op 1.18
phase0 processRegistryUpdates - 250000 normalcase 10.723 us/op 11.290 us/op 0.95
phase0 processRegistryUpdates - 250000 badcase_full_deposits 477.12 us/op 454.13 us/op 1.05
phase0 processRegistryUpdates - 250000 worstcase 0.5 154.76 ms/op 112.40 ms/op 1.38
altair processRewardsAndPenalties - 250000 normalcase 56.681 ms/op 45.606 ms/op 1.24
altair processRewardsAndPenalties - 250000 worstcase 45.722 ms/op 55.275 ms/op 0.83
phase0 getAttestationDeltas - 250000 normalcase 6.2055 ms/op 6.6142 ms/op 0.94
phase0 getAttestationDeltas - 250000 worstcase 8.7788 ms/op 7.1375 ms/op 1.23
phase0 processSlashings - 250000 worstcase 82.052 us/op 90.993 us/op 0.90
altair processSyncCommitteeUpdates - 250000 117.09 ms/op 114.25 ms/op 1.02
BeaconState.hashTreeRoot - No change 372.00 ns/op 400.00 ns/op 0.93
BeaconState.hashTreeRoot - 1 full validator 101.34 us/op 99.395 us/op 1.02
BeaconState.hashTreeRoot - 32 full validator 1.2810 ms/op 1.2990 ms/op 0.99
BeaconState.hashTreeRoot - 512 full validator 13.787 ms/op 11.733 ms/op 1.18
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 140.15 us/op 115.72 us/op 1.21
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 2.5710 ms/op 1.5978 ms/op 1.61
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 26.168 ms/op 29.077 ms/op 0.90
BeaconState.hashTreeRoot - 1 balances 93.242 us/op 105.56 us/op 0.88
BeaconState.hashTreeRoot - 32 balances 1.1703 ms/op 1.0607 ms/op 1.10
BeaconState.hashTreeRoot - 512 balances 10.916 ms/op 10.568 ms/op 1.03
BeaconState.hashTreeRoot - 250000 balances 176.88 ms/op 198.88 ms/op 0.89
aggregationBits - 2048 els - zipIndexesInBitList 10.594 us/op 10.883 us/op 0.97
byteArrayEquals 32 64.959 ns/op 65.235 ns/op 1.00
Buffer.compare 32 38.545 ns/op 38.035 ns/op 1.01
byteArrayEquals 1024 1.7397 us/op 1.7410 us/op 1.00
Buffer.compare 1024 48.183 ns/op 44.052 ns/op 1.09
byteArrayEquals 16384 26.270 us/op 27.811 us/op 0.94
Buffer.compare 16384 199.66 ns/op 227.19 ns/op 0.88
byteArrayEquals 123687377 220.59 ms/op 207.50 ms/op 1.06
Buffer.compare 123687377 7.9889 ms/op 3.8142 ms/op 2.09
byteArrayEquals 32 - diff last byte 61.865 ns/op 61.457 ns/op 1.01
Buffer.compare 32 - diff last byte 38.379 ns/op 37.816 ns/op 1.01
byteArrayEquals 1024 - diff last byte 1.8080 us/op 1.7162 us/op 1.05
Buffer.compare 1024 - diff last byte 47.933 ns/op 45.785 ns/op 1.05
byteArrayEquals 16384 - diff last byte 28.487 us/op 27.584 us/op 1.03
Buffer.compare 16384 - diff last byte 218.04 ns/op 207.27 ns/op 1.05
byteArrayEquals 123687377 - diff last byte 214.38 ms/op 204.99 ms/op 1.05
Buffer.compare 123687377 - diff last byte 7.8626 ms/op 5.9454 ms/op 1.32
byteArrayEquals 32 - random bytes 6.0580 ns/op 4.7510 ns/op 1.28
Buffer.compare 32 - random bytes 42.880 ns/op 43.743 ns/op 0.98
byteArrayEquals 1024 - random bytes 5.6900 ns/op 4.7270 ns/op 1.20
Buffer.compare 1024 - random bytes 42.035 ns/op 38.155 ns/op 1.10
byteArrayEquals 16384 - random bytes 6.6170 ns/op 4.9570 ns/op 1.33
Buffer.compare 16384 - random bytes 41.032 ns/op 40.818 ns/op 1.01
byteArrayEquals 123687377 - random bytes 18.300 ns/op 8.4600 ns/op 2.16
Buffer.compare 123687377 - random bytes 53.390 ns/op 41.250 ns/op 1.29
regular array get 100000 times 43.884 us/op 42.245 us/op 1.04
wrappedArray get 100000 times 52.595 us/op 42.612 us/op 1.23
arrayWithProxy get 100000 times 10.809 ms/op 10.620 ms/op 1.02
ssz.Root.equals 57.799 ns/op 58.258 ns/op 0.99
byteArrayEquals 56.995 ns/op 55.227 ns/op 1.03
Buffer.compare 11.828 ns/op 9.6700 ns/op 1.22
shuffle list - 16384 els 4.7376 ms/op 4.7282 ms/op 1.00
shuffle list - 250000 els 70.126 ms/op 69.112 ms/op 1.01
processSlot - 1 slots 19.540 us/op 17.592 us/op 1.11
processSlot - 32 slots 4.3260 ms/op 3.6608 ms/op 1.18
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 50.934 ms/op 53.320 ms/op 0.96
getCommitteeAssignments - req 1 vs - 250000 vc 2.2961 ms/op 2.2331 ms/op 1.03
getCommitteeAssignments - req 100 vs - 250000 vc 3.4491 ms/op 3.3350 ms/op 1.03
getCommitteeAssignments - req 1000 vs - 250000 vc 3.7328 ms/op 3.6079 ms/op 1.03
findModifiedValidators - 10000 modified validators 477.44 ms/op 400.04 ms/op 1.19
findModifiedValidators - 1000 modified validators 394.94 ms/op 319.39 ms/op 1.24
findModifiedValidators - 100 modified validators 340.83 ms/op 330.01 ms/op 1.03
findModifiedValidators - 10 modified validators 328.29 ms/op 296.11 ms/op 1.11
findModifiedValidators - 1 modified validators 317.82 ms/op 302.66 ms/op 1.05
findModifiedValidators - no difference 321.37 ms/op 312.43 ms/op 1.03
compare ViewDUs 4.4158 s/op 4.2675 s/op 1.03
compare each validator Uint8Array 1.8158 s/op 1.8654 s/op 0.97
compare ViewDU to Uint8Array 1.1921 s/op 1.3267 s/op 0.90
migrate state 1000000 validators, 24 modified, 0 new 790.74 ms/op 716.96 ms/op 1.10
migrate state 1000000 validators, 1700 modified, 1000 new 967.05 ms/op 1.0339 s/op 0.94
migrate state 1000000 validators, 3400 modified, 2000 new 1.1805 s/op 1.3037 s/op 0.91
migrate state 1500000 validators, 24 modified, 0 new 744.80 ms/op 715.79 ms/op 1.04
migrate state 1500000 validators, 1700 modified, 1000 new 942.11 ms/op 1.0640 s/op 0.89
migrate state 1500000 validators, 3400 modified, 2000 new 1.1640 s/op 1.3849 s/op 0.84
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 5.3800 ns/op 5.5800 ns/op 0.96
state getBlockRootAtSlot - 250000 vs - 7PWei 844.14 ns/op 952.58 ns/op 0.89
computeProposers - vc 250000 7.2872 ms/op 7.2513 ms/op 1.00
computeEpochShuffling - vc 250000 68.674 ms/op 71.061 ms/op 0.97
getNextSyncCommittee - vc 250000 127.91 ms/op 133.71 ms/op 0.96
computeSigningRoot for AttestationData 25.525 us/op 27.532 us/op 0.93
hash AttestationData serialized data then Buffer.toString(base64) 1.3058 us/op 1.3210 us/op 0.99
toHexString serialized data 813.75 ns/op 868.66 ns/op 0.94
Buffer.toString(base64) 173.91 ns/op 213.98 ns/op 0.81

by benchmarkbot/action

g11tech
g11tech previously requested changes Jan 29, 2024
View reviewed changes
Copy link
Contributor

@g11tech g11tech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets observe sim tests reliability, and then review this if things on sim tests are stable (blocking it till then)

@nazarhussain nazarhussain force-pushed the nh/sim-merge-test branch 3 times, most recently from 123ba02 to 2f0e55d Compare February 16, 2024 10:47
@nazarhussain
Copy link
Contributor Author

Tried locally and on CI many consecutive runs. With recent changes made to improve sim tests, it's stability seem good now.
https://github.com/ChainSafe/lodestar/actions/workflows/test-sim.yml
image

@wemeetagain @g11tech I feel it's good to review/merge this PR.

@nflaig
Copy link
Member

nflaig commented Feb 16, 2024

Sim merge tests seem to be stuck, running for almost 3 hours already

@wemeetagain wemeetagain merged commit e9a3f07 into unstable Feb 26, 2024
18 of 20 checks passed
@wemeetagain wemeetagain deleted the nh/sim-merge-test branch February 26, 2024 16:10
@nflaig nflaig mentioned this pull request Feb 27, 2024
Open
14 tasks
@wemeetagain
Copy link
Member

🎉 This PR is included in v1.17.0 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wemeetagain wemeetagain wemeetagain approved these changes

@g11tech g11tech g11tech left review comments

Assignees

@nazarhussain nazarhussain

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@nazarhussain @nflaig @wemeetagain @g11tech