Update libsecp256k1 (endomorphism, test improvements) #20147

sipa · 2020-10-14T18:50:36Z

This updates the libsecp256k1 subtree to the latest master, which includes:

Enabling the GLV endomorphism optimization by default (and removing support for the non-GLV EC multiplication)
Added a proof for the correctness of the lambda split algorithm by roconnor-blockstream (other code was relying on the fact that it always outputs 128 bit results, which isn't at all obvious).
Improved exhaustive tests, in particular for the Schnorr signature module
Various other testing and CI improvements

c6b6b8f Merge bitcoin#830: Rip out non-endomorphism code + dependencies c582aba Consistency improvements to the comments 63c6b71 Reorder comments/function around scalar_split_lambda 2edc514 WNAF of lambda_split output has max size 129 4232e5b Rip out non-endomorphism code ebad841 Check correctness of lambda split without -DVERIFY fe7fc1f Make lambda constant accessible 9d2f2b4 Add tests to exercise lambda split near bounds 9aca2f7 Add secp256k1_split_lambda_verify acab934 Detailed comments for secp256k1_scalar_split_lambda 76ed922 Increase precision of g1 and g2 6173839 Switch to our own memcmp function 63150ab Merge bitcoin#827: Rename testrand functions to have test in name c5257ae Merge bitcoin#821: travis: Explicitly set --with-valgrind bb1f542 Merge bitcoin#818: Add static assertion that uint32_t is unsigned int or wider a45c1fa Rename testrand functions to have test in name 5006895 Merge bitcoin#808: Exhaustive test improvements + exhaustive schnorrsig tests 4eecb4d travis: VALGRIND->RUN_VALGRIND to avoid confusion with WITH_VALGRIND 66a765c travis: Explicitly set --with-valgrind d7838ba Merge bitcoin#813: Enable configuring Valgrind support 7ceb0b7 Merge bitcoin#819: Enable -Wundef warning 8b7dcdd Add exhaustive test for extrakeys and schnorrsig 08d7d89 Make pubkey parsing test whether points are in the correct subgroup 87af00b Abstract out challenge computation in schnorrsig 63e1b2a Disable output buffering in tests_exhaustive.c 39f67dd Support splitting exhaustive tests across cores e99b26f Give exhaustive_tests count and seed cmdline inputs 49e6630 refactor: move RNG seeding to testrand b110c10 Change exhaustive test groups so they have a point with X=1 cec7b18 Select exhaustive lambda in function of order 78f6cdf Make the curve B constant a secp256k1_fe d7f39ae Delete gej_is_valid_var: unused outside tests 8bcd78c Make secp256k1_scalar_b32 detect overflow in scalar_low c498366 Move exhaustive tests for recovery to module be31791 Make group order purely compile-time in exhaustive tests e73ff30 Enable -Wundef warning c0041b5 Add static assertion that uint32_t is unsigned int or wider 4ad408f Merge bitcoin#782: Check if variable=yes instead of if var is set in travis.sh 412bf87 configure: Allow specifying --with[out]-valgrind explicitly 34debf7 Modify .travis.yml to explictly pass no in env vars instead of setting to nothing a0e99fc Merge bitcoin#814: tests: Initialize random group elements fully 5738e86 tests: Initialize random group elements fully c9939ba Merge bitcoin#812: travis: run bench_schnorrsig a51f2af travis: run bench_schnorrsig ef37761 Change travis.sh to check if variables are equal to yes instead of not-empty. Before this, setting `VALGRIND=wat` was considered as true, and to make it evaluate as false you had to unset the variable `VALGRIND=` but not it checks if `VALGRIND=yes` and if it's not `yes` then it's evaluated to false git-subtree-dir: src/secp256k1 git-subtree-split: c6b6b8f

jonatack · 2020-10-14T20:52:53Z

Concept ACK.

benthecarman · 2020-10-14T21:06:48Z

ACK 9e5626d

$  ./test/lint/git-subtree-check.sh src/secp256k1
src/secp256k1 in HEAD currently refers to tree c79797189781d6656eba6cb155c249c57b4f5c9a
src/secp256k1 in HEAD was last updated in commit 52380bf304b1c02dda23f1e2fad0159e29b2f7a2 (tree c79797189781d6656eba6cb155c249c57b4f5c9a)
src/secp256k1 in HEAD was last updated to upstream commit c6b6b8f1bb044d7d1aa065ebb674adde98a36a8e (tree c79797189781d6656eba6cb155c249c57b4f5c9a)
GOOD

fanquake

ACK 9e5626d - performed a squash and checked that the changes were the same. The non-endomorphism code has now been ripped out.

how-to-verify-subtree notes here: https://github.com/fanquake/core-review/blob/master/subtree-merge.md

fanquake · 2020-10-15T03:18:26Z

@jamesob this would be a good PR for you to throw into bitcoinperf.

jamesob · 2020-10-16T13:40:13Z

Ran a local IBD comparison on this for 8000 blocks (height 500_000 - 508_000) and to my surprise didn't see any performance difference. Compared the HEAD of this PR to the commit in master preceding it (c2c4dba). Could be that this platform is bottlenecked by IO and not CPU?

commands index

bench name	command
ibd.local.range.500000.508000	`bitcoind -dbcache=300 -debug=coindb -debug=bench -listen=0 -connect=0 -addnode=127.0.0.1:8888 -prune=9999999 -printtoconsole=0 -assumevalid=000000000000000000176c192f42ad13ab159fdb20198b87e7ba3c001e47b876`

#20147 vs. `c2c4dba` (absolute)

bench name	x	#20147	`c2c4dba`
ibd.local.range.500000.508000.total_secs	2	737.8656 (± 3.1841)	738.3541 (± 7.6443)
ibd.local.range.500000.508000.peak_rss_KiB	2	1746294.0000 (± 2362.0000)	1756260.0000 (± 3168.0000)

#20147 vs. `c2c4dba` (relative)

bench name	x	#20147	`c2c4dba`
ibd.local.range.500000.508000.total_secs	2	1	1.001
ibd.local.range.500000.508000.peak_rss_KiB	2	1	1.006

sipa · 2020-10-16T16:07:20Z

@jamesob That's certainly possible. With such a low dbcache, it's also possible it's just spending a lot of time in in preparing leveldb updates to flush.

…ents) 52380bf Squashed 'src/secp256k1/' changes from 8ab24e8..c6b6b8f (Pieter Wuille) Pull request description: This updates the libsecp256k1 subtree to the latest master, which includes: * Enabling the GLV endomorphism optimization by default (and removing support for the non-GLV EC multiplication) * Added a proof for the correctness of the lambda split algorithm by roconnor-blockstream (other code was relying on the fact that it always outputs 128 bit results, which isn't at all obvious). * Improved exhaustive tests, in particular for the Schnorr signature module * Various other testing and CI improvements ACKs for top commit: fanquake: ACK 9e5626d - performed a squash and checked that the changes were the same. The non-endomorphism code has now been ripped out. benthecarman: ACK 9e5626d Tree-SHA512: 50fda5f3f934ee525f01cfc15e4f5efbc5261a97f2b77fe1b3453ee0edcf1281ad74ab4532a2fe1fe907652dd47023beff8cf3d73bf34f65ac914a694b9e7110

Sjors · 2020-10-16T18:07:51Z

Your assumevalid block is at height 522,000 so you're not checking signatures, I think.

jamesob · 2020-10-17T17:22:36Z

Great call @Sjors, that should've occurred to me (d'oh). In any case, here are some much less surprising results showing improvement after disabled assumevalid.

commands index

bench name	command
ibd.local.range.500000.508000	`bitcoind -dbcache=300 -debug=coindb -debug=bench -listen=0 -connect=0 -addnode=127.0.0.1:8888 -prune=9999999 -printtoconsole=0 \-assumevalid=0`

#20147 vs. `c2c4dba` (absolute)

bench name	x	#20147	`c2c4dba`
ibd.local.range.500000.508000.total_secs	2	1529.8273 (± 5.1097)	1780.5817 (± 5.7187)
ibd.local.range.500000.508000.peak_rss_KiB	2	1718228.0000 (± 17852.0000)	1686864.0000 (± 6596.0000)

#20147 vs. `c2c4dba` (relative)

bench name	x	#20147	`c2c4dba`
ibd.local.range.500000.508000.total_secs	2	1.000	1.164
ibd.local.range.500000.508000.peak_rss_KiB	2	1.019	1.000

For kicks, here's the utxo cache flush history (I need to normalize the times in this graph...).

fanquake · 2020-10-19T01:00:48Z

Thanks @jamesob 👍

Update secp256k1 again This migrates us to the same dependency version that upstream Bitcoin Core migrated to in bitcoin/bitcoin#20147.

…ents) 52380bf Squashed 'src/secp256k1/' changes from 8ab24e8..c6b6b8f (Pieter Wuille) Pull request description: This updates the libsecp256k1 subtree to the latest master, which includes: * Enabling the GLV endomorphism optimization by default (and removing support for the non-GLV EC multiplication) * Added a proof for the correctness of the lambda split algorithm by roconnor-blockstream (other code was relying on the fact that it always outputs 128 bit results, which isn't at all obvious). * Improved exhaustive tests, in particular for the Schnorr signature module * Various other testing and CI improvements ACKs for top commit: fanquake: ACK 9e5626d - performed a squash and checked that the changes were the same. The non-endomorphism code has now been ripped out. benthecarman: ACK 9e5626d Tree-SHA512: 50fda5f3f934ee525f01cfc15e4f5efbc5261a97f2b77fe1b3453ee0edcf1281ad74ab4532a2fe1fe907652dd47023beff8cf3d73bf34f65ac914a694b9e7110

sipa added 2 commits October 14, 2020 11:41

Update libsecp256k1 subtree to latest master

9e5626d

DrahtBot added Build system Docs labels Oct 14, 2020

fanquake approved these changes Oct 15, 2020

View reviewed changes

fanquake merged commit f2e6d14 into bitcoin:master Oct 15, 2020

maflcko removed the Docs label Oct 15, 2020

str4d mentioned this pull request Oct 16, 2020

Update secp256k1 zcash/zcash#4757

Merged

str4d mentioned this pull request Oct 21, 2020

Update secp256k1 again zcash/zcash#4804

Merged

zkbot added a commit to zcash/zcash that referenced this pull request Oct 25, 2020

Auto merge of #4804 - str4d:update-secp256k1-again, r=daira

fef4b91

Update secp256k1 again This migrates us to the same dependency version that upstream Bitcoin Core migrated to in bitcoin/bitcoin#20147.

pinheadmz mentioned this pull request Nov 6, 2020

Use endomorphism in libsecp256k1 for performance improvement bcoin-org/bcoin#1003

Open

bitcoin locked as resolved and limited conversation to collaborators Feb 15, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update libsecp256k1 (endomorphism, test improvements) #20147

Update libsecp256k1 (endomorphism, test improvements) #20147

sipa commented Oct 14, 2020 •

edited by fanquake

Loading

jonatack commented Oct 14, 2020

benthecarman commented Oct 14, 2020 •

edited

Loading

fanquake left a comment

fanquake commented Oct 15, 2020

jamesob commented Oct 16, 2020

sipa commented Oct 16, 2020

Sjors commented Oct 16, 2020

jamesob commented Oct 17, 2020

fanquake commented Oct 19, 2020

Update libsecp256k1 (endomorphism, test improvements) #20147

Update libsecp256k1 (endomorphism, test improvements) #20147

Conversation

sipa commented Oct 14, 2020 • edited by fanquake Loading

jonatack commented Oct 14, 2020

benthecarman commented Oct 14, 2020 • edited Loading

fanquake left a comment

Choose a reason for hiding this comment

fanquake commented Oct 15, 2020

jamesob commented Oct 16, 2020

commands index

#20147 vs. c2c4dba (absolute)

#20147 vs. c2c4dba (relative)

sipa commented Oct 16, 2020

Sjors commented Oct 16, 2020

jamesob commented Oct 17, 2020

commands index

#20147 vs. c2c4dba (absolute)

#20147 vs. c2c4dba (relative)

fanquake commented Oct 19, 2020

sipa commented Oct 14, 2020 •

edited by fanquake

Loading

benthecarman commented Oct 14, 2020 •

edited

Loading

#20147 vs. `c2c4dba` (absolute)

#20147 vs. `c2c4dba` (relative)

#20147 vs. `c2c4dba` (absolute)

#20147 vs. `c2c4dba` (relative)