ETH RPC issues #12205
Comments
The nodes running are full nodes (snap synced). curl -X POST -H "Content-Type: application/json" \
--data '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":["latest", false],"id":1}' \
http://localhost:1234/rpc/v1
# Same block is returned when node is stuck
time curl -X POST -H "Content-Type: application/json" \
--data '{"method":"eth_getLogs","params":[{
"fromBlock": "0x3E31B0",
"toBlock": "0x3E31BB",
"address":["0xa02cbf1dc75058cc6f2f5b8e7a9087425f5248e3","0x0E4a2276Ac259CF226eEC6536f2b447Fc26F2D8a"],
"topics":[["0x47cee97cb7acd717b3c0aa1435d004cd5b3c8c57d70dbceb4e4458bbd60e39d4","0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925"]]
}],"id":1,"jsonrpc":"2.0"}' \
http://localhost:1234/rpc/v1
{"id":1,"jsonrpc":"2.0","result":[]}
# Different response times are `5.966s` and `20.225s`
time curl -X POST -H "Content-Type: application/json" \
--data '{"method":"eth_getLogs","params":[{
"fromBlock": "0x3bc92e",
"toBlock": "0x3bc92f",
"address":["0xa02cbf1dc75058cc6f2f5b8e7a9087425f5248e3","0x0E4a2276Ac259CF226eEC6536f2b447Fc26F2D8a"],
"topics":[["0x47cee97cb7acd717b3c0aa1435d004cd5b3c8c57d70dbceb4e4458bbd60e39d4","0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925"]]
}],"id":1,"jsonrpc":"2.0"}' \
http://localhost:1234/rpc/v1
{"id":1,"jsonrpc":"2.0","result":[]}
curl -X POST -H "Content-Type: application/json" --data 0.00s user 0.01s system 0% cpu 3:40.03 total
Major ETH RPC APIs used in subgraphs/watchers are:
|
|
Thank you for all the details.
|
|
One of the lotus snap synced nodes just got stuck (doesn't move past block 4079492): lotus version 1.27.1-rc2+mainnet+git.82e35e9b4 Recent logs from the lotus terminal: |
|
Lotus node version 1.28.0-rc4+mainnet+git.b5aab0121 is pretty much unusable at this point as we keep running into the error in point 4 here: #12205 (comment) Switching over to 1.27 to keep the watchers running. |
|
@ashwinphatak We just got a fix rolled out for 4 at #12212 |
|
One of the upgraded Lotus nodes also got stuck It is stuck at block Recent logs from the lotus terminal: |
|
Posting logs from 2 nodes that got stuck at block
|
|
@nikugogoi we have 2 go-libp2p upgrades in |
|
#12261 should help with the performance problems in |
|
There's a lot reported and being tracked in this issue. I'm hoping to update on where we're at. This is my understanding/guess, but I'd appreciate confirmation/correction from @ashwinphatak, @aarshkshah1992, or @rvagg
|
|
|
@aarshkshah1992 @BigLep @rvagg We ran a snap-synced node with version: $ lotus version
Daemon: 1.29.0-rc1+mainnet+git.5bd97edb4+api1.5.0
Local: lotus version 1.29.0-rc1+mainnet+git.5bd97edb4And tried out some
Summary:
The performance seems to have improved overall (query times reduced from
After keeping the node running for 4 days, we haven't seen much of a performance drop for
We are yet to try with an archive node using the rc release. |
|
We ran an archive node with the latest version: $ lotus version
Daemon: 1.29.0+mainnet+git.1ff3b360b+api1.5.0
Local: lotus version 1.29.0+mainnet+git.1ff3b360bAnd tried out some
Yes, after upgrading, the performance has improved for archive node as well. |
|
Thanks for reporting back @prathamesh0 . @prathamesh0 or @rvagg : is there other outstanding work we need to track with this issue or are we good to close this? (There are additional items actively being worked on currently to further improve RPC performance like #12388 ). |
|
We're continuing to do adjacent work but it seems that the original concerns raised here have been addressed. We could leave this open for general collection but I'd prefer to have separate issues as things raised. @prathamesh0 if you don't mind, could we ask you to open a new issue when you have specific items. We don't mind how many or how often you create, and if the template is too much, just skip most of it, or even just start from a blank issue (which I always do tbh: https://github.com/filecoin-project/lotus/issues/new). Happy to have additional feedback and specific things to work on. Thanks for the info so far! |
|
Thanks, closing this issue for now. We'll create new issues as suggested for further feedback. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Checklist
Lotus component
Lotus Version
Repro Steps
No response
Describe the Bug
fromortoare a null block (doesn't seem to happen with all nodes, needs more analysis)Tooling
Configuration Options
export LOTUS_CHAINSTORE_ENABLESPLITSTORE="true"
export LOTUS_EVENTS_ENABLEACTOREVENTSAPI="true"
export LOTUS_INDEX_ENABLEMSGINDEX="true"
export LOTUS_FEVM_ENABLEETHRPC="true"
export LOTUS_FEVM_ENABLEETHHASHTOFILECOINCIDMAPPING="true"
./bin/lotus daemon
The text was updated successfully, but these errors were encountered: