Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cache /get_transactions call on lastblocks collector #4

Open
cirocosta opened this issue Aug 3, 2021 · 2 comments
Open

cache /get_transactions call on lastblocks collector #4

cirocosta opened this issue Aug 3, 2021 · 2 comments

Comments

@cirocosta
Copy link
Owner

depending on the scraping interval, it can be somewhat expensive on monerod to serve the requests it takes from monero-exporter.

it we look at the lastblocks collector's fetchData (when it reaches out to monerod), we can note the call to /get_transactions passing the hash of every transaction included in that block:

txnsResp, err := c.client.GetTransactions(ctx, blockJSON.TxHashes)
if err != nil {
return fmt.Errorf("get txns: %w", err)
}

as we expect blocks to arrive every 2min, a prometheus instance configured with a polling interval of 15s would be making 8 times more calls to that endpoint than it really should (as they'll all return the same answer).

in that case, because the call /get_block is made using a hash (rather than a height), we could cache the response from get_transactions for such block, thus reducing the load on monerod.

BEFORE

        on every scrape:

                lastBlockHeader := get_lastblock_header()
                lastBlock := get_block(lastBlockHeader.hash)
                txns := get_transactions(lastBlock.hashes)


AFTER

        on every scrape:

                lastBlockHeader := get_lastblock_header()

                cacheEntry, found := blocksCache.get(lastBlockHeader.hash)
                if found {
                        // we're good
                        return
                }

                lastBlock := get_block(lastBlockHeader.hash)
                txns := get_transactions(lastBlock.hashes)
@cirocosta
Copy link
Owner Author

cirocosta commented Aug 3, 2021

note that get_transactions is indeed solely used by the lastblock collector - transactionpool does not need it as the call to get_transaction_pool already returns the detailed information we need.

looking at rpc_access_tracking, we can tell get_transaction is indeed quite expensive atm

 ~ $ monero daemon rpc-access-tracking
RPC                             COUNT   TIME SPENT SERVING      CREDITS
get_version                     90      499.904µs               0
hard_fork_info                  46      611.606µs               46
get_alternate_chains            1       952.902µs               0
rpc_access_info                 416     1.807211ms              0
get_public_nodes                1       3.617268ms              0
get_txpool_backlog              81      11.612045ms             81
get_block_header_by_height      154     15.306969ms             154
get_block_headers_range         58      17.226973ms             58
get_net_stats                   12434   23.167619ms             0
get_outs                        21      31.506649ms             161
getblockcount                   8578    37.103013ms             0
get_outs_bin                    13      85.465299ms             3149
sync_info                       10      91.58133ms              20
get_bans                        8577    118.654689ms            0
send_raw_tx                     12      175.204242ms            1200
get_height                      484     183.888138ms            0
rpc_access_tracking             14847   346.920509ms            0
get_hashes                      281     2.024486379s            5886
get_output_distribution_bin     13      2.359060579s            260
get_base_fee_estimate           8604    4.66403806s             8604
get_last_block_header           28780   46.608057363s           28780
get_peer_list                   12561   1m10.117860338s         0
get_block                       20159   1m31.193389823s         20159
get_blocks                      99032   2m30.302909893s         137586
get_transaction_pool_stats      20121   4m3.598594881s          20121
get_connections                 31671   4m27.280631191s         0
get_transaction_pool            11606   5m5.373659997s          513958      <<<
get_transaction_pool_hashes     98092   8m3.274738272s          131568
get_info                        65858   10m3.184014312s         65858
get_transactions                93120   37m27.277965493s        592453      <<<

even though when we divide count by time, get_transaction_pool reveals to be more expensive (which sure, makes sense)

get_transactions       24135840.310619477 
get_transaction_pool   26308944.62566707

I don't think we can do much about get_transaction_pool though

@cirocosta
Copy link
Owner Author

how much of a big deal this is? well, not a lot

Screen Shot 2021-08-03 at 8 47 38 AM

but, there's clearly an opportunity for improvements here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant