debug_traceBlockByHash returns "method handler crashed" for certain blocks

[tibineacsu95]

Hello,

We have been running multiple Avalanche Archive nodes (both mainnet and testnet) for quite some time but lately we've had some issues showing up.

For Mainnet:

Example RPC Call:

curl http://127.0.0.1:9650/ext/bc/C/rpc -X POST -H "Content-Type: application/json" --data '{"method":"debug_traceBlockByHash","params":["0xef836031411b2c15ef330ce28911aaf6de1995ccbdf45419fbfd4c43aa992a3e", {"tracer": "callTracer", "tracerConfig": {"onlyTopCall":false,"withLog":true}}],"id":1,"jsonrpc":"2.0"}'

Returns:

{"jsonrpc":"2.0","id":1,"error":{"code":-32603,"message":"method handler crashed"}}

For Testnet:

Example RPC Call:

curl https://127.0.0.1:9650/ext/bc/C/rpc -X POST -H "Content-Type: application/json" --data '{"method":"debug_traceBlockByHash","params":["0x260238a8b034c48d2b48f40c3a34587077aefc012a96abd026e116c594c0a688", {"tracer": "prestateTracer", "tracerConfig": {"onlyTopCall": false, "diffMode": true}}],"id":1,"jsonrpc":"2.0"}'

Returns:

{"jsonrpc":"2.0","id":1,"error":{"code":-32603,"message":"method handler crashed"}}

To give some context about the node setup environment, we are running the following binary version: avalanche/1.10.3 [database=v1.4.5, rpcchainvm=26, commit=eb6e7973a900dc647d29a2da8e7a233ac0a5fe77] and we have synced the nodes from scratch, given the fact that there were no official snapshots available at that time.

We were wondering whether anyone else has seen this problem before or not. Any ideas would be highly appreciated. Thanks in advance!

[StephenButtolph]

This seems to share the same root cause as: #1647. The fix for this should be staged for v1.10.4

[tibineacsu95]

Hey @StephenButtolph. Thank you for the quick response. Do you have an estimate on when the next version is scheduled to be released? After taking a quick look on #1647, I guess rolling back to v1.10.2 is also an option, but if v1.10.4 is coming in the next days, I guess it would be safer to just wait for it?

[StephenButtolph]

I'd expect v1.10.4 to be released either today or tomorrow

[tibineacsu95]

Thanks a lot for the details, Stephen! Have a great day!

[StephenButtolph]

Seems like it might slip another day (don't want you to be waiting for a fix tonight if it slips haha)

[tibineacsu95]

Sure thing. Thank you for the update! Really appreciate it!

[tibineacsu95]

Hello @StephenButtolph! We've updated a few of our nodes to v1.10.4 and the problem is still reproducible.

curl http://127.0.0.1:9650/ext/bc/C/rpc -X POST -H "Content-Type: application/json" --data '{"method":"debug_traceBlockByHash","params":["0xef836031411b2c15ef330ce28911aaf6de1995ccbdf45419fbfd4c43aa992a3e", {"tracer": "callTracer", "tracerConfig": {"onlyTopCall":false,"withLog":true}}],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"error":{"code":-32603,"message":"method handler crashed"}}

./avalanche-node/avalanchego --version
avalanche/1.10.4 [database=v1.4.5, rpcchainvm=26, commit=3e8552d072f0f1b8d29fcf3ec71f043f52d2c8dd]

I have also tried downgrading one of the nodes to v1.10.2 to see if the problem persists, but the call responds as below:

curl http://127.0.0.1:9650/ext/bc/C/rpc -X POST -H "Content-Type: application/json" --data '{"method":"debug_traceBlockByHash","params":["0xef836031411b2c15ef330ce28911aaf6de1995ccbdf45419fbfd4c43aa992a3e", {"tracer": "callTracer", "tracerConfig": {"onlyTopCall":false,"withLog":true}}],"id":1,"jsonrpc":"2.0"}'
{"jsonrpc":"2.0","id":1,"result":[{"result":{"type":"CREATE","from":"0xc873c70deff6f2e05e9709da70c910a127d5f107","to":"0xe3411df534f389ceb9394a495a6e64f73deb875e","value":"0x16345785d8a0000","gas":"0x20bf6","gasUsed":"0x20bf6","input":"0xa2a1623d000000000000000000000000000000000000000000000000016345785d8a00000000000000000000000000000000000000000000000000000000000000000080000000000000000000000000c873c70deff6f2e05e9709da70c910a127d5f107000000000000000000000000000000000000000000000000000000006093973c0000000000000000000000000000000000000000000000000000000000000002000000000000000000000000b31f66aa3c1e785363f0875a1b74e27b85fd66c70000000000000000000000006e7f5c0b9f4432716bdd0a77a3601291b9d9e985","error":"stack underflow (0 \u003c=\u003e 4)"}}]}

./avalanche-node/avalanchego --version
avalanche/1.10.2 [database=v1.4.5, rpcchainvm=26, commit=f7307d5fd015e75d6b40d26713c5fb6576dc6ffb]

Any advice on how to fix this? Thank you!

[aaronbuchwald]

Thanks for re-opening and sorry we mistakenly thought this was the same issue.

Looked into this with @morrisettjohn and we have a unit test reproducing the behavior now. Going to work on cleaning it up and putting together the fix. Thanks for your patience!

[tibineacsu95]

No worries at all! Glad you guys managed to reproduce it and are working on a fix. Really appreciate the update!

[aaronbuchwald]

Closing in favor of ava-labs/coreth#282 because this is an issue within Coreth.

If it's easier to follow it here, just re-open the issue with a message and I'll post an update here as well.

Thanks for reporting the issue!