Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BIP352 silentpayments module #1519

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

josibake
Copy link
Member

@josibake josibake commented Apr 19, 2024

This PR adds a new Silent Payments (BIP352) module to secp256k1. It is a continuation of the work started in #1471.

The module implements the full protocol, except for transaction input filtering and silent payment address encoding / decoding as those will be the responsibility of the wallet software. It is organized with functions for sending (prefixed with _sender) and receiving (prefixed by _recipient).

For sending

  1. Collect private keys into two lists: taproot_seckeys and plain_seckeys
    Two lists are used since the taproot_seckeys may need negation. taproot_seckeys are passed as keypairs to avoid the function needing to compute the public key to determine parity. plain_seckeys are passed as just secret keys
  2. Create the _silentpayment_recipient objects
    These structs hold the scan and spend public key and an index for remembering the original ordering. It is expected that a caller will start with a list of silent payment addresses (with the desired amounts), convert these into an array of recipients and then match the generated outputs back to the original silent payment addresses. The index is used to return the generated outputs in the original order
  3. Call silentpayments_sender_create_outputs to generate the xonly public keys for the recipients
    This function can be called with one or more recipients. The same recipient may be repeated to generate multiple outputs for the same recipient

For scanning

  1. Collect the public keys into two lists taproot_pubkeys and plain_pubeys
    This avoids the caller needing to convert taproot public keys into compressed public keys (and vice versa)
  2. Compute the input data needed, i.e. sum the public keys and compute the input_hash
    This is done as a separate step to allow the caller to reuse this output if scanning for multiple scan keys. It also allows a caller to use this function for aggregating the transaction inputs and storing them in an index to vend to light clients later (or for faster rescans when recovering a wallet)
  3. Call silentpayments_recipient_scan_outputs to scan the transaction outputs and return the tweak data (and optionally label information) needed for spending later

In addition, a few utility functions for labels are provided for the recipient for creating a label tweak and tweaked spend public key for their address. Finally, two functions are exposed in the API for supporting light clients, _recipient_created_shared_secret and _recipient_create_output_pubkey. These functions enable incremental scanning for scenarios where the caller does not have access to the transaction outputs:

  1. Calculating a shared secret
    This is done as a separate step to allow the caller to reuse the shared secret result when creating outputs and avoid needing to do a costly ECDH every time they need to check for an additional output
  2. Generate an output (with k = 0)
  3. Check if the output exists in the UTXO set (using their preferred light client protocol)
  4. If the output exists, proceed by generating a new output from the shared secret with k++

See examples/silentpayments.c for a demonstration of how the API is expected to be used.

Note for reviewers

My immediate goal is to get feedback on the API so that I can pull this module into bitcoin/bitcoin#28122 (silent payments in the bitcoin core wallet). That unblocks from finishing the bitcoin core PRs while work continues on this module.

Notable differences between this PR and the previous version

See #1427 and #1471 for discussions on the API design. This iteration of the module attempts to be much more high level and incorporate the feedback from #1471. I also added a secp256k1_silentpayments_public_data opaque data type, which contains the summed public key and the input_hash. My motivation here was:

  1. I caught myself mixing up the order of arguments between A_sum and recipient_spend_key, which was impossible to catch with ARG_CHECKS and would result in the scanning process finishing without errors, but not finding any outputs
  2. Combining public key and input_hash into the same data type allows for completely hiding input_hash from the caller, which makes for an overall simpler API IMO

I also removed the need for the recipient to generate a shared secret before using the secp256k1_silentpayments_recipient_scan_outputs function and instead create the shared secret inside the function.

Outstanding work

  • clean up the testing code
  • improve test coverage (currently only using the BIP352 test vectors)
  • optimize the implementation, where possible

Copy link
Contributor

@theStack theStack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Concept ACK

Left some initial feedback, especially around the scanning routine, will do an in-depth review round soon. Didn't look closer at the public_data type routines and the examples yet.

@josibake josibake force-pushed the bip352-silentpayments-module branch from 3d08027 to 8b48bf1 Compare April 24, 2024 08:38
@josibake
Copy link
Member Author

Rebased on #1518 (3d08027 -> 8b48bf1, compare)

@josibake josibake force-pushed the bip352-silentpayments-module branch from 8b48bf1 to f5585d4 Compare April 24, 2024 09:38
@josibake
Copy link
Member Author

Updated 8b48bf1 -> f5585d4 (bip352-silentpayments-module-rebase -> bip352-silentpayments-module-02, compare):

  • Fix function documentation for _recipient_scan_outputs
  • Replace VERIFY_CHECK with return 0; in _sender_create_outputs
  • Remove unneeded declassify code from _sender_create_outputs
  • Change _gej_add_ge to _gej_add_var in _recipient_public_data_create
  • Fix label scanning in _recipient_scan_outputs
  • Remove unneeded prints from the tests

For the label scanning, I looked for an example of using an invalid public key but didn't see anything except for the invalid_pubkey_bytes in the tests. For now, if the output is found without a label, I'm setting found_with_label = 0 and saving the found output in both the output and label field. Happy to change this if there is a better suggestion for communicating an invalid public key.

I also used secp256k1_pubkey_save instead of output = *tx_outputs, as I think this makes the code more clear.

Copy link
Contributor

@theStack theStack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second review round through, looks good so far! Left a bunch of nits, mostly about naming and missing ARG_CHECKS etc.

Comment on lines 164 to 175
} else {
printf("Failed to create keypair\n");
return 1;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: i think the else-brach can be removed (or at least, the return 1; in it); if keypair creation fails, we want to continue in the loop and try with another secret key

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really done?

edit: Well, we should change the other examples and perhaps also the API docs. Creating keys in a while loop is not good practice. If you hit an invalid key, you're most probably not very lucky, but very unlucky because your randomness is broken. But hey, yeah, let's just yolo and try again. :)

So I guess either is fine for now: keep the loop for consistency with the other examples, or just return 1 here, but having a loop and the else branch is certainly a smell.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops! I think I had a local commit for this but forgot to squash it in.

Regarding best practices, creating the keys in a loop seemed excessive to me but figured I'd just copy the existing examples in case there was something I wasn't understanding.

Considering this is new code, seems fine to me to break from the other examples and then I can open a separate PR to clean up the examples/API docs.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually fixed this time to take out the while loop and return if it fails to create the key.

Copy link
Contributor

@real-or-random real-or-random Jul 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see #1570

@josibake josibake force-pushed the bip352-silentpayments-module branch 2 times, most recently from 9d75190 to 1a3a00b Compare May 3, 2024 08:21
@josibake
Copy link
Member Author

josibake commented May 3, 2024

Thanks for the thorough review, @theStack ! I've addressed your feedback, along with some other changes.


Update f5585d4 -> 1a3a00b (bip352-silentpayments-module-02 -> bip352-silentpayments-module-03, compare)

  • Spelling and wording cleanups, notably:
    • s/receiver/recipient/, s/labeled/labelled/
    • s/scan_seckey/scan_key/
  • Reduce duplicate code in scan_outputs
  • Add ARG_CHECKs
  • Update tests
  • Add benchmark for scan_outputs

The sending tests now check that the generated outputs match exactly one of the possible expected output sets. Previously, the sending tests were checking that the generated outputs exist in the array of all possible outputs, but this wouldn't catch a bug where k is not being set correctly e.g. [Ak=0, Bk=0] would (incorrectly) pass [Ak=0, Bk=1, Ak=1, Bk=0] but will now (correctly) fail [[Ak=0, Bk=1], [Ak=1, Bk=0]]

@josibake josibake force-pushed the bip352-silentpayments-module branch from 1a3a00b to 92f5920 Compare May 3, 2024 11:11
@josibake
Copy link
Member Author

josibake commented May 3, 2024

@josibake josibake force-pushed the bip352-silentpayments-module branch from 92f5920 to 56ed901 Compare May 8, 2024 12:48
@josibake
Copy link
Member Author

josibake commented May 8, 2024

Copy link
Contributor

@theStack theStack left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went through another round. To the best of my knowledge, this PR matches the BIP352 specification and I'm close to non-cryptographer-light-ACKing it :-)

Found some nits an one open TODO that should probably be discussed though.

@josibake josibake force-pushed the bip352-silentpayments-module branch from 56ed901 to bd66eaa Compare May 31, 2024 12:34
@josibake
Copy link
Member Author

Rebased on master to fix merge conflict 56ed901 -> bd66eaa (bip352-silentpayments-module-04-rebase -> bip352-silentpayments-module-05-rebase, compare)

@josibake
Copy link
Member Author

CI failure seems related to not being able to install valgrind via homebrew and unrelated to my change so ignoring for now (cc @real-or-random for confirmation?).

@josibake josibake force-pushed the bip352-silentpayments-module branch from bd66eaa to 2dde8f1 Compare May 31, 2024 13:37
@josibake
Copy link
Member Author

Thanks for the review @theStack ! Sorry for the slow response, I somehow missed the notification for your review 😅


Update bd66eaa -> 2dde8f1 (bip352-silentpayments-module-05-rebase -> bip352-silentpayments-module-06, compare)

  • spelling, grammar, and fixups per @theStack 's review
  • Added ARG_CHECKs to check for the sum of the private keys / public keys being zero

Per #1519 (comment), I agree returning 0 is not the right thing to do, but having multiple error codes also seemed gross. I think an ARG_CHECK makes sense here because if the caller passed all valid seckeys / pubkeys and then they sum to zero, in principle its the caller passing incorrect arguments. The only thing the caller can do at this point is try again with different arguments. For the sender, this would mean repeating coin selection to get a different input set, and for the recipient this would mean skipping the transaction and moving on to the next one. Also happy to change if there is a better suggestion!

@real-or-random
Copy link
Contributor

CI failure seems related to not being able to install valgrind via homebrew and unrelated to my change so ignoring for now (cc @real-or-random for confirmation?).

Indeed, see #1536

@real-or-random
Copy link
Contributor

real-or-random commented May 31, 2024

Some general notes

On error handling in general

Error handling is hard, and the caller usually can't really recover from an error anyway. This is in particular true on malicious inputs: there's no reason to try to continue dealing with the attacker, and you simply want to abort. That's why, as a general rule, we try to avoid error paths as much as possible. This usually boils down to merging all errors into a single one, i.e., a) have just a single error "code" for all possible errors, b) and in the case of a multi-stage thing involving multiple function calls, have just a single place where errors are returned.

Signature verification is a good example. A (signature, message, pubkey) triple is either valid or not. The caller should not care why exactly a signature fails to verify, so we don't even want to expose this to the caller.

However, signature verification this is also a nice example of a case in which we stretch the rules a bit. Signature verification is implemented as two-stage process: 1. Parse the public key (which can fail). 2. Check the signature (which can fail). Purely from a "safe" API point of view, this is not great because we give the user two functions and two error paths instead of one. Ideally, there could just be one verification function which also takes care of parsing (this is how it's defined BIP340). The primary reason why we want to have a separate parsing function in this case is performance: if you check several signatures under the same key, you don't want to parse, which involves computing the y-coordinate, every time.

ARG_CHECK

ARG_CHECK will call the "illegal (argument) callback", which, by default, crashes. See the docs here:

/** Set a callback function to be called when an illegal argument is passed to
The callback/crash indicates to the caller that there's a bug in the caller's code.

What does this mean for this discussion?

  • Added ARG_CHECKs to check for the sum of the private keys / public keys being zero

Per #1519 (comment), I agree returning 0 is not the right thing to do, but having multiple error codes also seemed gross. I think an ARG_CHECK makes sense here because if the caller passed all valid seckeys / pubkeys and then they sum to zero, in principle its the caller passing incorrect arguments. The only thing the caller can do at this point is try again with different arguments. For the sender, this would mean repeating coin selection to get a different input set, and for the recipient this would mean skipping the transaction and moving on to the next one. Also happy to change if there is a better suggestion!

So let's take a look at the two sides:

On the sender side: The secret keys sum up to zero (sum_i a_i = 0)

This will happen only with negligible probability for honestly generated (=random) secret keys. That is, this will in practice only happen if the caller has a bug, or the caller has been tricked into using these secret keys, e.g., if someone else has crafted malicious secret keys for the caller. Since the latter is not a crazy scenario, we should not use ARG_CHECK here.

We can just return 0 here to indicate to the caller that we can't continue with these function inputs. And even if there are other error cases, I don't see a reason why the caller code should care much about why the function fails. As long as you call the function with honestly generated inputs, everything will work out. (Devs will be interested in the exact failure case when debugging the caller's code, but I think they can figure out during debugging then. "Normal" caller code should get just a single error code.)

On the recipient side: The public keys sum up to infinity (sum_i A_i = 0) [1]

Again, this can only happen if the sender is malicious. But since we're not the sender, it's entirely possible that the sender is malicious. And then these inputs are certainly legal, they're just not valid. (In the same sense as it's perfectly legal to use the signature verification algorithm on an invalid signature.) So an ARG_CHECK will not be appropriate at all: a malicious sender could trigger it and crash the scanning process.

We should also simply return 0 to indicate that this transaction is not well-formed/not eligible for SP. And again, even if there are other error cases, I don't see a reason why the caller should care why this transaction is not eligible.

Alternatively, we could even return 1, store infinity in the public_data, and simply make sure that scanning won't find any payments in that case. This would avoid the error path for this function entirely. But if the caller then calls secp256k1_silentpayments_recipient_create_shared_secret, I think we'd just postpone the error to this function, and for this function, I don't see another way than returning an error. So I'm not convinced that this is better.

[1] We should perhaps rename "infinity" to "zero"... ;)

@josibake
Copy link
Member Author

josibake commented May 31, 2024

@real-or-random thanks for the response, this is super helpful.

Devs will be interested in the exact failure case when debugging the caller's code, but I think they can figure out during debugging then

In hindsight, I think my preference for ARG_CHECK was "better error messages as to what went wrong," but I now realize it was because I was thinking as a dev ;). Also an oversight on my part: I didn't realize/forgot that ARG_CHECK is actually crashing the program by default. I certainly agree that we don't want this in either failure case.

Alternatively, we could even return 1, store infinity in the public_data, and simply make sure that scanning won't find any payments in that case. This would avoid the error path for this function entirely. But if the caller then calls secp256k1_silentpayments_recipient_create_shared_secret, I think we'd just postpone the error to this function, and for this function, I don't see another way than returning an error. So I'm not convinced that this is better.

If we imagine an index + light client scenario, the public_data would be created by the index and then sent to the light client, where the light client would call secp256k1_silentpayments_recipient_create_shared_secret (and then get the error). Given this, I think it would be better to have the error path so that the index ends up not storing any data at all for the malicious crafted transaction, which saves space for the index and bandwidth for the light client.


Thinking about this a bit more:

That's why, as a general rule, we try to avoid error paths as much as possible. This usually boils down to merging all errors into a single one, i.e., a) have just a single error "code" for all possible errors, b) and in the case of a multi-stage thing involving multiple function calls, have just a single place where errors are returned.

Most of the high-level functions in our API are calling multiple lower-level functions and so far the approach has been something like:

if (!secp256k1_func_that_returns_0_on_error(args)) {
    return 0;
}
...
if (!secp256k1_another_func_that_returns_0_on_error(args)) {
    return 0;
}

Perhaps its worth looking to consolidate and try and only return an error at the end of a multi-stage process? This would mean ignoring the return values for a lot of the lower level function calls, which initially made me feel a bit weird. But in light of your recent comment, feels like this might be the preferred approach?

EDIT: reading your comment again, I realize "error paths" is not really talking about branches in the code and more error paths for the user.

@theStack
Copy link
Contributor

theStack commented Jun 1, 2024

We should also simply return 0 to indicate that this transaction is not well-formed/not eligible for SP. And again, even if there are other error cases, I don't see a reason why the caller should care why this transaction is not eligible.

Makes sense. My worry was that without an explicit error-code for this corner case, some users wouldn't even be aware of an indirect "not eligible" case and more likely interpret a return value of 0 as "only possible if there's a logic error on our side, so let's assert for success" (given the passed in data is public and already verified for consensus-validity). But in the end that's more a matter of good API documentation I guess.

An example for the "input public keys sum up to point of infinity" case ($\sum_i A_i = 0$) is now available on the Signet chain via tx d73f4a19f3973e90af6df62e735bb7b31f3d5ab8e7e26e7950651b436d093313 [1], mined in block 198023. It consists of two inputs spending P2WPKH prevouts with negated pubkeys $(x,y)$ and $(x,-y)$ (easy to verify by looking at the second item of the witness stack each, where only the first byte for encoding the sign bit differs), and one dummy P2TR output. It hopefully helps SP implementations to identify potential problems with this corner case early. As first example and proof that it triggers the discussed code path, it makes the Silent Payment Index PR #28241 crash, which asserts on a return value of 1 for _recipient_public_data_create.

I think it would be also a good idea to add this scenario to the BIP352 test vectors, or at least a unit test in this PR?

[1] created with the following Python script: https://github.com/theStack/bitcoin/blob/202405-contrib-bip352_input_pubkeys_cancelled/contrib/silentpayments/submit_input_pubkeys_infinity_tx.py

* any further elliptic-curve operations from the wallet.
*/

/* This struct serves as an In param for passing the silent payment address
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/* This struct serves as an In param for passing the silent payment address
/* This struct serves as an input argument for passing the silent payment address

would be more consistent with secp256k1.h

Comment on lines 33 to 37
* When more than one recipient is used the recipient array will be sorted in
* place as part of generating the outputs, but the generated outputs will be
* returned in the original ordering specified by the index to ensure the
* caller is able to match up the generated outputs to the correct silent
* payment address (e.g. to be able to assign the correct amounts to the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* When more than one recipient is used the recipient array will be sorted in
* place as part of generating the outputs, but the generated outputs will be
* returned in the original ordering specified by the index to ensure the
* caller is able to match up the generated outputs to the correct silent
* payment address (e.g. to be able to assign the correct amounts to the
* When more than one recipient is used, the recipient array will be sorted in
* place as part of generating the outputs, but the generated outputs will be
* returned in the original ordering specified by the index to ensure the
* caller is able to match up the generated outputs to the correct silent
* payment address (e.g., to be able to assign the correct amounts to the

Comment on lines 407 to 411
* Being a light client, Carol likely does not have access to the
* transaction outputs. This means she will need to first generate
* an output, check if it exists in the UTXO set (e.g. BIP158 or
* some other means of querying) and only proceed to check the next
* output (by incrementing `k`) if the first output exists.
Copy link

@Sosthene00 Sosthene00 Oct 26, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems that this description of the light client workflow would only work for users that don't use labels at all. While we could assume that indeed most light client users wouldn't use labels, I think we'd rather make a slighter more general example here assuming that Carol also use labels, or if we don't want to make the example more complicated at the very least explain it here in comments. Here's a quick rewriting of this paragraph based on how we made it work for Dana:

/** Being a light client, Carol likely does not have access to the
* transaction outputs. This means she will need to first generate
* the first outputs she would get in a transaction, meaning the output 
* from her plain spend_pubkey + one for each label she monitors 
* with _k_ == 0 , and check if it exists in the UTXO set (e.g. BIP158 or 
* some other means of querying).
*  
* As soon as she finds an output she can request the whole transaction
* (or rather the whole block to not reveal which transactions 
* she's interested in) and use the same function than Bob from there 
* to find any other output that belongs to her.
*  
* She would then repeat this operation for each `public_data` she received
*/

Alternatively I guess she could also increment k when she has a match and keep looking for outputs this way, but she needs to generate the k+1 output for each label also. Maybe that would indeed make more sense if we go for UTXO set scanning instead of scanning each block sequentially.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a good call out. I added a sentence to indicate that, while its not recommended, Carol can still uses labels as a light client using the addition method.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think another reason I didn't include light client labels is because this would require a function for adding arbitrary public keys, which at this point we are trying to avoid.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I agree it's kind of an edge case and we want to be careful not to compromise too much because of it, but on the other hand we should acknowledge it because it will definitely happen I think. I'll try to complete the tests on that part and see how bad it makes things

@josibake
Copy link
Member Author

josibake commented Nov 7, 2024

Big thanks to all the review over the last few months and apologies for my slow response. I'm in the process of rebasing and incorporating review feedback, should have the PR updated shortly.

* parameter pairs, depending on whether the seckeys correspond to x-only
* outputs or not.
*
* Returns: 1 if creation of outputs was successful. 0 if an error occured.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9d6769f: typo nit: s/occured/occurred here and in few more places in the file

hash->bytes = 64;
}

static void secp256k1_silentpayments_create_t_k(secp256k1_scalar *t_k_scalar, const unsigned char *shared_secret33, unsigned int k) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9d6769f: micro nit: might make sense to add const qualifier to the int too. also for the int in other functions - secp256k1_silentpayments_recipient_create_output_pubkey, secp256k1_silentpayments_recipient_create_label_tweak, secp256k1_silentpayments_create_output_pubkey

/* Calculate the input hash and tweak a_sum, i.e., a_sum_tweaked = a_sum * input_hash */
secp256k1_silentpayments_calculate_input_hash(input_hash, outpoint_smallest36, &A_sum_ge);
secp256k1_scalar_set_b32(&input_hash_scalar, input_hash, &overflow);
ret &= !overflow;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9d6769f: BIP sounds like it ignores overflow in https://github.com/bitcoin/bips/blob/master/bip-0352.mediawiki#creating-outputs. though everywhere in the code input_hash overflow would lead to failure. It's a very unlikely scenario and not sure which option is preferable, but would be nice to keep BIP and code consistent.

secp256k1_write_be32(k_serialized, k);
secp256k1_sha256_write(&hash, k_serialized, sizeof(k_serialized));
secp256k1_sha256_finalize(&hash, hash_ser);
secp256k1_scalar_set_b32(t_k_scalar, hash_ser, NULL);
Copy link
Contributor

@stratospher stratospher Dec 3, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9d6769f: BIP says to fail If t_k is not valid tweak - https://github.com/bitcoin/bips/blob/master/bip-0352.mediawiki#creating-outputs
here, we're ignoring the overflow. we could keep it consistent with the BIP?

(have a similar comment in call site of secp256k1_silentpayments_create_t_k)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially, I believe the thinking was not having untestable branches (e.g., not able to find a preimage which causes an overflow), but I think the point you make regarding the BIP is a good one: better to stick to the specification to avoid a scenario where one implementation handles the overflow and another does not. Will update.

ARG_CHECK(output33 != NULL);
ARG_CHECK(public_data != NULL);
/* Only allow public_data to be serialized if it has the hash and the summed public key
* This helps protect against accidentally serialiazing just a the summed public key A
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

94c6e1f: typo nit:

Suggested change
* This helps protect against accidentally serialiazing just a the summed public key A
* This helps protect against accidentally serializing just the summed public key A

ret &= secp256k1_silentpayments_recipient_public_data_load_pubkey(ctx, &pubkey, public_data);
secp256k1_silentpayments_recipient_public_data_load_input_hash(input_hash, public_data);
ret &= secp256k1_ec_pubkey_tweak_mul(ctx, &pubkey, input_hash);
secp256k1_ec_pubkey_serialize(ctx, output33, &pubkeylen, &pubkey, SECP256K1_EC_COMPRESSED);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

94c6e1f: micro nit: could add VERIFY_CHECK(pubkeylen == 33)

CHECK(secp256k1_silentpayments_recipient_public_data_parse(CTX, &pd, malformed) == 0);

/* This public_data object was created with combined = 0, i.e., it has both the input hash and summed public keypair.
* In instances where the caller has access the the full transaction, they should use `_scan_outputs` instead, so
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

94c6e1f:

Suggested change
* In instances where the caller has access the the full transaction, they should use `_scan_outputs` instead, so
* In instances where the caller has access to the full transaction, they should use `_scan_outputs` instead, so

return 0;
}
combined = (int)public_data->data[0];
if (!combined) {
Copy link
Contributor

@stratospher stratospher Dec 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

94c6e1f: I think comments here would be useful! (thanks @theStack for explaining the intuition behind this!)

maybe something like:

/**
    * If combined is set, pubkey in `public_data` contains input_hash * pubkey. Save an extra point multiplication
    * by only having to compute shared_secret = recipient_scan_key * (input_hash * pubkey).
    * 
    * If combined is not set, update recipient_scan_key to contain recipient_scan_key * input_hash and then compute
    * shared_secret = (recipient_scan_key * input_hash) * pubkey.
*/

EDIT: part I got confused with was I thought everyone could do (recipient_scan_key * input_hash) but light clients wouldn't be able to since they don't have access to summed pubkey, input_hash(computed from summed pubkey). would have liked to see that part in the comments but also think it doesn't fit here since it's bitcoin specific.

k = 0;
for (i = 0; i < n_recipients; i++) {
if ((i == 0) || (secp256k1_ec_pubkey_cmp(ctx, &last_recipient.scan_pubkey, &recipients[i]->scan_pubkey) != 0)) {
/* If we are on a different scan pubkey, its time to recreate the the shared secret and reset k to 0.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9d6769f:

Suggested change
/* If we are on a different scan pubkey, its time to recreate the the shared secret and reset k to 0.
/* If we are on a different scan pubkey, its time to recreate the shared secret and reset k to 0.

secp256k1_silentpayments_create_t_k(&t_k_scalar, shared_secret, k);

/* Calculate P_output = B_spend + t_k * G
* This can fail if t_k overflows the curver order, but this is statistically improbable
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

94c6e1f: it don't think it can fail because of overflow at this point (the overflow is ignored and t_k has already been converted to a scalar in secp256k1_silentpayments_create_t_k). we could move this comment up into secp256k1_silentpayments_create_t_k and handle failure inside that function.


/* Compute input private keys sum: a_sum = a_1 + a_2 + ... + a_n */
a_sum_scalar = secp256k1_scalar_zero;
for (i = 0; i < n_plain_seckeys; i++) {
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How should we handle secret keys that add up to SECP256k1_N?
I tested this locally with ALICE_SECKKEY = 0xeadc78165ff1f8ea94ad7cfdc54990738a4c53f6e0507b42154201b8e5dff3b1 and another secret key I generated with SECP256K1_N - ALICE_SECKEY = 0x152387e9a00e07156b5283023ab66f8b306288efcef824f9aa905cd3ea564d90. Passing these two plain secret keys causes secp256k1_silentpayments_sender_create_outputs to fail

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had an offline conversation with @josibake. The conclusion is that there is nothing to do. The sender has to try again with a different set of pubkeys

Copy link

@Eunovo Eunovo Dec 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@josibake Here's a diff with the test case for this edge case,
https://github.com/josibake/secp256k1/compare/9d6769f4..144584b3

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for writing a test! However, I believe this case is already covered with the ORDERC malformed key test case. Instead of adding keys that sum, we just pass in a key that is already >= the curve order.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this case is already covered with the ORDERC malformed key test case

I checked this. IIUC, the function exits earlier when you use a malformed key. Using keys that sum to SECP256K1_N uses a different code path.
There's a test for keys that sum to zero / point at infinity included in the test vectors so we still don't need to add one. I verified this by checking the coverage using

cmake -B build-cov -DCMAKE_BUILD_TYPE=Coverage
cmake --build build-cov -j 6
build-cov/bin/noverify_tests
find build-cov -name '*.gcda'|xargs gcov -t > output.gcov

secp256k1_pubkey *label,
unsigned char *label_tweak32,
const unsigned char *recipient_scan_key,
unsigned int m
Copy link

@Eunovo Eunovo Dec 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

9d6769f4: m could be const

int main(void) {
enum { N_INPUTS = 2, N_OUTPUTS = 3 };
unsigned char randomize[32];
unsigned char xonly_print[32];
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

5c546e2: nit: Can we change xonly_print to serialized_xonly_pk? or maybe add a comment /* Serialized xonly pk */

Copy link
Contributor

@real-or-random real-or-random Dec 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that _print is a bit redundant, but shouldn't unsigned char ... [32] make it clear that this is the serialization?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To you and me, yes, but to the uninitiated, spelling it out in the variable name might help. This code is meant to be read by anyone trying to use the silent-payments API, so it helps to make things clearer for beginners. That said, I don't have any strong feelings about this; it's just a "nit."

@real-or-random
Copy link
Contributor

We discussed this in the meeting today.

@josibake Can you summarize what the status of the PR is? We're considering doing a sprint week around mid-March if that fits your (and other contributors') schedules and the status of the PR.

@josibake
Copy link
Member Author

Hey @real-or-random, current status when I left off in early December was finishing a rebase and responding to outstanding feedback. I'll be picking up that work next week with a goal of having this ready for review and all outstanding feedback addressed by beginning of February. I think a sprint in mid March works perfectly for me! I'll be babysitting this PR through February so its ready for March and will also have time in March to respond to reviewer feedback.

theStack and others added 10 commits February 11, 2025 15:27
Add a routine for the entire sending flow which takes a set of private keys,
the smallest outpoint, and list of recipients and returns a list of
x-only public keys by performing the following steps:

1. Sum up the private keys
2. Calculate the input_hash
3. For each recipient group:
    3a. Calculate a shared secret
    3b. Create the requested number of outputs

This function assumes a single sender context in that it requires the
sender to have access to all of the private keys. In the future, this
API may be expanded to allow for a multiple senders or for a single
sender who does not have access to all private keys at any given time,
but for now these modes are considered out of scope / unsafe.

Internal to the library, add:

1. A function for creating shared secrets (i.e., a*B or b*A)
2. A function for generating the "SharedSecret" tagged hash
3. A function for creating a single output public key
Add function for creating a label tweak. This requires a tagged hash
function for labels. This function is used by the receiver for creating
labels to be used for a) creating labelled addresses and b) to populate
a labels cache when scanning.

Add function for creating a labelled spend pubkey. This involves taking
a label tweak, turning it into a public key and adding it to the spend
public key. This function is used by the receiver to create a labelled
silent payment address.

Add tests for the label API.
Add routine for scanning a transaction and returning the necessary
spending data for any found outputs. This function works with labels via
a lookup callback and requires access to the transaction outputs.
Requiring access to the transaction outputs is not suitable for light
clients, but light client support is enabled by exposing the
`_create_shared_secret` and `_create_output_pubkey` functions in the
API. This means the light client will need to manage their own scanning
state, so wherever possible it is preferrable to use the
`_recipient_scan_ouputs` function.

Add an opaque data type for passing around the summed input public key (A_sum)
and the input hash tweak (input_hash). This data is passed to the scanner
before the ECDH step as two separate elements so that the scanner can
multiply b_scan * input_hash before doing ECDH.

Add functions for deserializing / serializing a public_data object to
and from a public key. When serializing a public_data object, the
input_hash is multplied into A_sum. This is so the object can be stored
as public key for wallet rescanning later, or to vend to light clients.
For the light client, a `_parse` function is added which parses the
compressed public key serialization into a `public_data` object.

Finally, add test coverage for the recieiving API.
Demonstrate sending, scanning, and light client scanning.
Add a benchmark for a full transaction scan and for scanning a single
output. Only benchmarks for scanning are added as this is the most
performance critical portion of the protocol.
Add the BIP-352 test vectors. The vectors are generated with a Python script
that converts the .json file from the BIP to C code:

$ ./tools/tests_silentpayments_generate.py test_vectors.json > ./src/modules/silentpayments/vectors.h
Co-authored-by: Jonas Nick <[email protected]>
Co-authored-by: Sebastian Falbesoner <[email protected]>
@josibake josibake force-pushed the bip352-silentpayments-module branch from f42e0dd to 71df073 Compare February 12, 2025 15:59
@josibake
Copy link
Member Author

A big thanks to all the reviewers and there patience waiting on me to respond and rebase 😅

I've added a commit for constant time tests (h/t to @jonasnick and @theStack!) and addressed a majority of the review comments. As some changes were necessary in order to pass the constant time tests, there are also some TODO comments which I will be addressing, along with the remaining PR feedback (pls don't take it personally if I miss a review comment; I am finding Github's UI exceedingly difficult to work with 😠 ).

@real-or-random , @jonasnick I left a TODO comment in the tests where I have two test cases for ensuring the _create_outputs function can handle malformed public keys passed in via the recipients arguments. Since this argument is provided by the user, I think it's reasonable to test for invalid public keys but I couldn't figure out how to do this with the way the CHECK_ILLEGAL macro works. I left a more detailed explanation in the TODO comment, would appreciate suggestions here!

* the recipients label cache during scanning.
*
* For creating the labels cache,
* `secp256k1_silentpayments_recipient_create_label_tweak` can be used.
Copy link

@nymius nymius Feb 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* `secp256k1_silentpayments_recipient_create_label_tweak` can be used.
* `secp256k1_silentpayments_recipient_create_label` can be used.

* the recipient uses labels. This allows for checking if a label exists in
* the recipients label cache and retrieving the label tweak during scanning.
*
* For the labels cache, `secp256k1_silentpayments_recipient_create_label_tweak`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* For the labels cache, `secp256k1_silentpayments_recipient_create_label_tweak`
* For the labels cache, `secp256k1_silentpayments_recipient_create_label`

if(DEFINED SECP256K1_ENABLE_MODULE_EXTRAKEYS AND NOT SECP256K1_ENABLE_MODULE_EXTRAKEYS)
message(FATAL_ERROR "Module dependency error: You have disabled the extrakeys module explicitly, but it is required by the silentpayments module.")
endif()
set(SECP256K1_ENABLE_MODULE_EXTRAKEYS ON)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IIUC, we also want to enable the schorrsig module here

Suggested change
set(SECP256K1_ENABLE_MODULE_EXTRAKEYS ON)
set(SECP256K1_ENABLE_MODULE_SCHNORRSIG ON)
set(SECP256K1_ENABLE_MODULE_EXTRAKEYS ON)

(that also matches the autotools-equivalent part)

* to identify the resulting transaction as a silent payments transaction and potentially link the transaction
* back to the silent payment address
*/
memset(hash_ser, 0, sizeof(hash_ser));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should use the dedicated memory cleanse function and also clear out the hash object

Suggested change
memset(hash_ser, 0, sizeof(hash_ser));
secp256k1_memclear(hash_ser, sizeof(hash_ser));
secp256k1_sha256_clear(&hash);

* could result in a third party being able to identify the transaction as a silent payments transaction
* and potentially link the transaction back to a silent payment address
*/
memset(&shared_secret, 0, sizeof(shared_secret));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
memset(&shared_secret, 0, sizeof(shared_secret));
secp256k1_memclear(&shared_secret, sizeof(shared_secret));

return 0;
}

/* Serialize B_m */
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/* Serialize B_m */

as the result is not serialized anymore (it was in an early version, IIRC)

* for later use, the caller can save the result with
* `silentpayments_recipient_public_data_serialize`.
*
* Returns: 1 if tweak data creation was successful. 0 if an error occured.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* Returns: 1 if tweak data creation was successful. 0 if an error occured.
* Returns: 1 if public data creation was successful. 0 if an error occured.

}
public_data->data[0] = 1;
secp256k1_ec_pubkey_serialize(ctx, &public_data->data[1], &pubkeylen, &pubkey, SECP256K1_EC_UNCOMPRESSED);
secp256k1_memclear(&public_data->data[1 + pubkeylen], 32);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: can just use the usual memset here, as this is no data on the stack and hence not at risk of optimized out (and it's not secret data anyway)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.