Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve the performance of the shielded sync ledger client #4016

Merged
merged 7 commits into from
Nov 12, 2024

Conversation

sug0
Copy link
Collaborator

@sug0 sug0 commented Nov 12, 2024

Describe your changes

The shielded sync's ledger client continues to perform poorly, even after #3578. This PR introduces some optimizations to the ledger client, and improves its UX a bit.

  • Fetches are now logged more frequently on the progress bar. Previously, they weren't being logged because concurrent fetches were starving each other on the first semaphore. To fix this, fetch_shielded_transfers now has a single semaphore point, at the top of the method, rather than two.
  • Errors returned from RPC servers are no longer printed. They weren't that helpful anyway, and were spamming the screen, obfuscating the output of the progress bars.
  • A linear backoff sleep has been added to the ledger client, on errors, to allow the client to recover from potential rate limiting/throttling imposed by the CometBFT RPC server. On successful fetches, the backoff is reduced, returning to the normal/optimal fetch speed.
  • The block batch size parameter has been exposed to the CLI and SDK, allowing users to specify the number of blocks that are fetched per concurrent fetch job. This may help performance, in some cases.

In general, the indexer masp client still performs about 10x better than the ledger masp client, while fetching shielded txs.

Checklist before merging

  • If this PR has some consensus breaking changes, I added the corresponding breaking:: labels
    • This will require 2 reviewers to approve the changes
  • If this PR requires changes to the docs or specs, a corresponding PR is opened in the namada-docs repo
    • Relevant PR if applies:
  • If this PR affects services such as namada-indexer or namada-masp-indexer, a corresponding PR is opened in that repo
    • Relevant PR if applies:

@sug0 sug0 requested review from batconjurer and grarco November 12, 2024 14:26
sug0 added a commit that referenced this pull request Nov 12, 2024
sug0 added a commit that referenced this pull request Nov 12, 2024
@sug0 sug0 force-pushed the tiago/ss-ledger-client-improvements branch from 605c1f5 to 1f85e3f Compare November 12, 2024 14:36
@sug0 sug0 force-pushed the tiago/ss-ledger-client-improvements branch from 1f85e3f to b2332bf Compare November 12, 2024 14:36
@sug0 sug0 added the UX label Nov 12, 2024
Copy link

codecov bot commented Nov 12, 2024

Codecov Report

Attention: Patch coverage is 66.12903% with 21 lines in your changes missing coverage. Please review.

Project coverage is 73.92%. Comparing base (732505c) to head (b2332bf).
Report is 8 commits behind head on main.

Files with missing lines Patch % Lines
crates/core/src/control_flow/time.rs 38.09% 13 Missing ⚠️
crates/sdk/src/masp/utilities.rs 80.48% 8 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4016      +/-   ##
==========================================
- Coverage   73.92%   73.92%   -0.01%     
==========================================
  Files         341      341              
  Lines      106467   106510      +43     
==========================================
+ Hits        78709    78740      +31     
- Misses      27758    27770      +12     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@sug0 sug0 requested a review from batconjurer November 12, 2024 16:55
@sug0 sug0 added the merge Ready to merge - mergifyio bot will add the PR to merge queue when all checks pass label Nov 12, 2024
@mergify mergify bot merged commit cb1e963 into main Nov 12, 2024
23 of 24 checks passed
@mergify mergify bot deleted the tiago/ss-ledger-client-improvements branch November 12, 2024 17:02
mergify bot pushed a commit that referenced this pull request Nov 12, 2024
(cherry picked from commit b2332bf)
mergify bot added a commit that referenced this pull request Nov 12, 2024
Improve the performance of the shielded sync ledger client (backport #4016)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking:SDK cli MASP merge Ready to merge - mergifyio bot will add the PR to merge queue when all checks pass SDK UX
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants