Left sig != right sig. Signature check failure. message occurs very frequently - follow up to 6288

[LaurenWags]

Description

While testing #6288 I noticed that I very frequently got the Left sig != right sig. Signature check failure. message in the terminal. This results in not all ballots being sent out, so not all BAT is properly distributed.

Steps to Reproduce

1. enable rewards with short contribution interval on staging. I also launched with --enable-logging=stderr --vmodule=*rewards*=6 to see log messages.
2. claim grant (30 BAT)
3. set AC amount to Up to 20 BAT
4. add verified and non verified publishers to AC table
5. add verified recurring tips in this order (5, 10, 10, 10, 1)
6. wait for monthly contribution to trigger
7. you should see 26 BAT goes through for monthly and 4 BAT from AC
8. Watch terminal.

• for simplicity, all verified publishers used belonged to the same publisher email

Actual result:

See several Left sig != right sig. Signature check failure. messages. Watch publisher account. Not all BAT is distributed as expected.

Expected result:

No Left sig != right sig. Signature check failure. messages

Reproduces how often:

easily

Brave version (brave://version info)

Brave	0.73.8 Chromium: 78.0.3904.50 (Official Build) nightly (64-bit)
Revision	2accdc52c79976e264cd2694df6db31d1fccd8e8-refs/branch-heads/3904@{#658}
OS	macOS Version 10.13.6 (Build 17G5019)

Version/Channel Information:

• Can you reproduce this issue with the current release? unsure
• Can you reproduce this issue with the beta channel? unsure
• Can you reproduce this issue with the dev channel? unsure
• Can you reproduce this issue with the nightly channel? yes

Other Additional Information:

• Does the issue resolve itself when disabling Brave Shields? n/a
• Does the issue resolve itself when disabling Brave Rewards? n/a
• Is the issue reproducible on the latest version of Chrome? n/a

Miscellaneous Information:

cc @NejcZdovc @rebron @brave/legacy_qa

[LaurenWags]

Ran some tests with just Auto Contribute and no recurring tips - did not see this message in the terminal. See brave/brave-core#3658 (comment) - but without recurring tips, all BAT expected was transferred to the publisher account.

[mandar-brave]

This is an issue observed before #6288 and has been on nightly since 0.68.
Likely impacting everyone.

cc @rebron

[NejcZdovc]

some more info about it

Left sig != right sig. Signature check failure.
02:59:32   
02:59:32  left 3i5HecEMhBDPUkIl1q6/4bEmmO17/FuawODhmx6Q2CR 8IB6ei3jUrm/gWWcquSy1B6AJ2UFPQJVQ5BQ7KY2E/m AdiauYiRCW81ABitpQzQGy+ZyjxtuvxVkwL1vWplOWU 6eFcOKcGZMcSn9KTTMqrhEFUHfCW6SVpnIUcUb7KvXQ AcE/5eiVGrXj5spee1f8StGDgHclgSlPZmbNb+xwUt2 2IbtraiAIbsMB449gt5RZpkNlRSKZk5tb91XIl0NI0G 5fmCzZ62FkiQlgdvbSa9OCiM2nptliESec3psfh7xE9 3woC807nH39Ybu8A/VM+389DvX1MZn0wSnDirZtZz5O 9cHfIXPEnWfE1pz31Hm4+knBJxdLn7OStCyC0owbN9k L1ZYM2Cs7w6mQWS4BPj+4bcaTJC43YJSPXrx9D+dNB 8IYsVVSbBdLsKIcCEjET8OC0J56QZGjUXxQOZdc0IrD 1VJTAzl9egs6EPI7w/Ps0G9U9qAb6Na8z/iTK8fkKtv
02:59:32  
02:59:32  right 5Wn11fx060n1v+s547jf3da1NZV6Ul5+kEMm2PTaU/2 8xxZxcoUnmp5hnaSSGKCJgU0vT+74rqn1kWhL8EDAPW 3nXuDanSqsyz/lXnhSTdzvdAxTiFnj1HYVRIOmCWI6H A4MdUpHLkLFccDykdbx40E/CPGqkNaTmPHVzbmA5XRI 6IjGKo7+Gn5sfpxTzITQfIgS1reVDET3Ht3smwTFRwV 4MWWTtuzpVhkWyr24ua3SORlEZc5HfQVQX/kj6dZkph 4QDuM0HvDRj4nwbbSwtFYIALLzgwcLIR5JMuQSE379D 7GJa/DxSIU/8P4cK356DWiUSzR0I1ELb5wQqwiRSHBh 9m0JqSaY6/rSVCyLr/BYC/1Aos2oj9XQpp0GA2dg+Ov BNjY4/f1JZ7L26wdw6SQl0ofB8hfn++t8DIl91Vzx3k AA/YaE8bjghz+LFevCr8pv00BrEOyts5yHaSdC4hlsX 2lWc4rHWXnVEUwlNtzK8rkCX9m0Ioh19cBxM0jvb3xM
02:59:32  Invalid signature in registerUserFinal 0c25b253059a42a828898e5d4013555
02:59:32  Cred.read: failed
02:59:32  Could not read ravk or cred. 
02:59:32  Cred.read: failed
02:59:32  Could not read ravk or cred. 
02:59:32  Cred.read: failed
02:59:32  Could not read ravk or cred. 
02:59:32  Cred.read: failed
02:59:32  Could not read ravk or cred. 

from https://staging.ci.brave.com/job/brave-browser-build-pr/job/native-button-styles/15/execution/node/658/log/

[kjozwiak]

As @NejcZdovc mentioned above, PR's are failing from time to time with Left sig != right sig. Signature check failure.. Examples:

• Add ability to publish GitHub pre-release as full release (uplift to 0.70.x) brave-core#3744 (comment)
• Add ability to publish GitHub pre-release as full release (uplift to 0.71.x) brave-core#3743 (comment)

[GeetaSarvadnya]

Reproduced in Windows 10 x64 - 0.70.120

[LaurenWags]

Worked with @NejcZdovc to confirm retries are working, so even though this error is fairly easily reproduced, the retry logic does kick in and the votes which failed are retried. Used the following STR to show this:

1. enable rewards with short contribution interval on staging. Launched with --enable-logging=stderr --vmodule=*rewards*=6 --rewards=staging=true,reconcile-interval=10,short-retries=true to see log messages, use staging env, set short contribution interval, and short retry interval.
2. Recovered wallet with 30 BAT
3. set AC amount to Up to 20 BAT
4. add verified and non verified publishers to AC table
5. add verified recurring tips in this order (5, 10, 10, 10, 1)
6. wait for monthly contribution to trigger
7. you should see 26 BAT goes through for monthly and 4 BAT from AC
8. Watch terminal and see Left sig != right sig. error messages
9. Continue to watch terminal to see retries occur.
   10 Also, check ledger_state to confirm. As retries are successful, the file gets smaller. Eventually, the batch section is empty, indicating that all retries were successful.

Confirmed above scenario using

Brave	0.70.121 Chromium: 78.0.3904.70 (Official Build) (64-bit)
Revision	edb9c9f3de0247fd912a77b7f6cae7447f6d3ad5-refs/branch-heads/3904@{#800}
OS	macOS Version 10.13.6 (Build 17G5019)

[jhoneycutt]

This appears to be caused by braveledger_contribution::PhaseTwo::PrepareBatchCallback overwriting an existing prepareBallot object with an object for a different viewingId. I believe that the check there for ballots[i].proofBallot_.empty() is a typo and should be ballots[i].prepareBallot_.empty()

[jhoneycutt]

A similar but separate issue exists in braveledger_contribution::PhaseTwo::ProofBatchCallback, filed as #6607.

[mandar-brave]

@jhoneycutt is there a fix being made available for this issue and #6607?

[mrose17]

@jhoneycutt - i'm trying to figure which line in phase_two.cc you are referring to in #6545 (comment) ... can you point me to the line in particular? thanks!

[jhoneycutt]

@mrose17 https://github.com/brave/brave-core/blob/master/vendor/bat-native-ledger/src/bat/ledger/internal/contribution/phase_two.cc#L292. However, fixing this typo isn't sufficient to fix the issue. We need to check that we are only assigning prepareBallot objects to ballots for the current viewingId.

[bsclifton]

Added to 0.74.x milestone, since this is merged 😄👍

[LaurenWags]

Verified on Nightly with

Brave	0.74.9 Chromium: 78.0.3904.87 (Official Build) nightly (64-bit)
Revision	20c21f4010010f32462ea8e1d6af30cef66d48c8-refs/branch-heads/3904@{#840}
OS	macOS Version 10.13.6 (Build 17G5019)

Verified test plan from brave/brave-core#3819 on Nightly 3x, however since amount of staging grants has changed (17.5 BAT instead of 30 BAT) the expected contributed amounts were different. I was only able to check that my own testing publisher accts got the expected BAT. Not all accounts used for testing were owned by me. However, during each test no left sig != right sig error was encountered.

[LaurenWags]

Verified passed with

Brave	0.71.113 Chromium: 78.0.3904.87 (Official Build) (64-bit)
Revision	20c21f4010010f32462ea8e1d6af30cef66d48c8-refs/branch-heads/3904@{#840}
OS	macOS Version 10.13.6 (Build 17G5019)

• Verified test plan from Left sig right sig match failure brave-core#3819 (restored a wallet with 30 BAT).
  • Confirmed expected BAT was sent to my publisher accts (unable to confirm all test accts, as not all pub accts belong to me).
  • Confirmed that the batch section of ledger_state was empty as per Left sig != right sig. Signature check failure. message occurs very frequently - follow up to 6288 #6545 (comment) .
  • Confirmed no Left sig != right sig message was observed.
• Verified test plan from Left sig right sig match failure brave-core#3819 except modified by using a staging grant for 17.5 BAT instead. In step 5, my tips were set up as 5, 10, 10, 1 due to lower wallet balance. In step 7, 16 BAT was contributed from monthly tips and 1.5 BAT from AC.
  • Confirmed expected BAT was sent to my publisher accts (unable to confirm all test accts, as not all pub accts belong to me).
  • Confirmed that the batch section of ledger_state was empty as per Left sig != right sig. Signature check failure. message occurs very frequently - follow up to 6288 #6545 (comment) .
  • Confirmed no Left sig != right sig message was observed.

Verification passed on

Brave	0.71.114 Chromium: 78.0.3904.97 (Official Build) (64-bit)
Revision	021b9028c246d820be17a10e5b393ee90f41375e-refs/branch-heads/3904@{#859}
OS	Windows 10 OS Version 1803 (Build 17134.1006)

• Verified test plan from Left sig right sig match failure brave-core#3819 except modified by using a staging grant for 17.5 BAT instead. In step 5, my tips were set up as 5, 10, 10, 1 due to lower wallet balance. In step 7, 16 BAT was contributed from monthly tips and 1.5 BAT from AC.
  • Confirmed expected BAT was sent to my publisher accts (unable to confirm all test accts, as not all pub accts belong to me).
  • Confirmed that the batch section of ledger_state was empty as per Left sig != right sig. Signature check failure. message occurs very frequently - follow up to 6288 #6545 (comment) .
  • Confirmed no Left sig != right sig message was observed.

Verification passed on

Brave	0.71.114 Chromium: 78.0.3904.97 (Official Build) (64-bit)
Revision	021b9028c246d820be17a10e5b393ee90f41375e-refs/branch-heads/3904@{#859}
OS	Ubuntu 18.04 LTS

• Verified test plan from Left sig right sig match failure brave-core#3819 except modified by using a staging grant for 17.5 BAT instead. In step 5, my tips were set up as 5, 10, 10, 1 due to lower wallet balance. In step 7, 16 BAT was contributed from monthly tips and 1.5 BAT from AC.
  • Confirmed expected BAT was sent to my publisher accts (unable to confirm all test accts, as not all pub accts belong to me).
  • Confirmed that the batch section of ledger_state was empty as per Left sig != right sig. Signature check failure. message occurs very frequently - follow up to 6288 #6545 (comment) .
  • Confirmed no Left sig != right sig message was observed.