Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic: deletion not supported #475

Closed
vrogojin opened this issue Oct 23, 2021 · 13 comments
Closed

panic: deletion not supported #475

vrogojin opened this issue Oct 23, 2021 · 13 comments
Labels
bug Something isn't working

Comments

@vrogojin
Copy link

Started fresh full node with snap sync
geth --syncmode=snap --config ./config.toml --datadir node --cache 18000 --rpc --http.port 9545 --http.corsdomain "*" --http.api "eth,web3,personal,net,txpool" --ws --ws.port 8546 --ws.origins "*" \ --ws.api "eth,web3,personal,net,txpool" --rpc.allow-unprotected-txs --txlookuplimit 0 console

After around an hour it crashes


INFO [10-23|00:29:36.318] Starting Geth on Ethereum mainnet...
INFO [10-23|00:29:36.321] Maximum peer count                       ETH=30 LES=0 total=30
INFO [10-23|00:29:36.321] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [10-23|00:29:36.321] Option nousb is deprecated and USB is deactivated by default. Use --usb to enable
Welcome to the Geth JavaScript console!

instance: Geth/v1.1.2-c4f93121-20210825/linux-amd64/go1.15.14
at block: 0 (Mon Apr 20 2020 21:46:54 GMT+0800 (+08))
 datadir: <my_data_dir_here>
 modules: admin:1.0 debug:1.0 eth:1.0 miner:1.0 net:1.0 parlia:1.0 personal:1.0 rpc:1.0 txpool:1.0 web3:1.0

To exit, press ctrl-d
> eth.syncing.
eth.syncing.constructor eth.syncing.toString eth.syncing.valueOf
> eth.syncing
{ 
  currentBlock: 0,
  highestBlock: 11997535,
  knownStates: 0,
  pulledStates: 0,
  startingBlock: 0
}
> net.peerCount
2
> net.peerCount
4
> panic: deletion not supported

goroutine 459309 [running]:
github.com/ethereum/go-ethereum/trie.(*StackTrie).TryUpdate(0xc069403520, 0xc077307a00, 0x20, 0x20, 0x2857fb0, 0x0, 0x0, 0xc019bb5698, 0xc07dd149a0)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:194 +0xd2
github.com/ethereum/go-ethereum/trie.(*StackTrie).Update(0xc069403520, 0xc077307a00, 0x20, 0x20, 0x2857fb0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:201 +0x77
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).processStorageResponse(0xc00000c780, 0xc061a85500)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:1928 +0x3f4
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).Sync(0xc00000c780, 0xd206470f7c921f7b, 0x3ee4d8465b3ed0b8, 0xb21e6d5f47bdd0a1, 0xa01669445816a58b, 0xc01662c780, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:653 +0xc2e
github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).run(0xc02de3d7c0)
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:318 +0x87
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).runStateSync
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:114 +0x1ef

Tried several times to do fresh sync.
Impossible to sync full node.
Any workaround?

Thanks
@vrogojin
Copy link
Author

After this crash, bash console freezes.
No I/O errors. No hardware errors.
What is going on?

@vrogojin
Copy link
Author

More clarifications:
System information
Geth
Version: 1.1.2 (binary pre-build as of 22.10.2021)
Architecture: amd64
Operating System: Ubuntu 20.04.3 LTS

Expected behaviour
No panic.

Actual behaviour
It panics.

Steps to reproduce the behaviour
Run fresh geth for an hour, then it panics.
geth --syncmode=snap --config ./config.toml --datadir node --cache 18000 --rpc --http.port 9545 --http.corsdomain "*" --http.api "eth,web3,personal,net,txpool" --ws --ws.port 8546 --ws.origins "*" \ --ws.api "eth,web3,personal,net,txpool" --rpc.allow-unprotected-txs --txlookuplimit 0 console

Backtrace


panic: deletion not supported

goroutine 459309 [running]:
github.com/ethereum/go-ethereum/trie.(*StackTrie).TryUpdate(0xc069403520, 0xc077307a00, 0x20, 0x20, 0x2857fb0, 0x0, 0x0, 0xc019bb5698, 0xc07dd149a0)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:194 +0xd2
github.com/ethereum/go-ethereum/trie.(*StackTrie).Update(0xc069403520, 0xc077307a00, 0x20, 0x20, 0x2857fb0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:201 +0x77
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).processStorageResponse(0xc00000c780, 0xc061a85500)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:1928 +0x3f4
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).Sync(0xc00000c780, 0xd206470f7c921f7b, 0x3ee4d8465b3ed0b8, 0xb21e6d5f47bdd0a1, 0xa01669445816a58b, 0xc01662c780, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:653 +0xc2e
github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).run(0xc02de3d7c0)
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:318 +0x87
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).runStateSync
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:114 +0x1ef

@grGred
Copy link

grGred commented Oct 26, 2021 •
edited
Loading

System information:
Geth
Version: 1.1.3
Architecture: amd64
Go Version: go1.13.8
Operating System: linux

Expected behaviour:
No panic. Countinues to synchronize.

Actual behaviour:
It panics, getting a bunch of errors and stops. Happens only with syncmode = snap.

Steps to reproduce the behaviour:
Run geth with command:
./build/bin/geth --config ./config.toml --datadir ./node --cache 32000 --rpc.allow-unprotected-txs --txlookuplimit 0 --maxpeers 4000 --syncmode snap --diffsync --cache.database 70 --cache.trie 30
In about an hour panic occurs, geth stops with output:

panic: deletion not supported

goroutine 217727 [running]:
github.com/ethereum/go-ethereum/trie.(*StackTrie).TryUpdate(0xc16e2e6ce0, 0xc12101a460, 0x20, 0x20, 0x29f31d8, 0x0, 0x0, 0x0, 0x0)
        github.com/ethereum/go-ethereum@/trie/stacktrie.go:194 +0xd0
github.com/ethereum/go-ethereum/trie.VerifyRangeProof(0x5615df180157484d, 0x7279ef9f81c8591d, 0xd838d87ee3b6e45c, 0xe96fc751a40f659a, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        github.com/ethereum/go-ethereum@/trie/proof.go:486 +0x20f
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).OnStorage(0xc01bba8780, 0x1c723e0, 0xc09c34ed80, 0x622732924331ee0b, 0xc067a6c580, 0x1d, 0x1d, 0xc067a6c840, 0x1d, 0x1d, ...)  
        github.com/ethereum/go-ethereum@/eth/protocols/snap/sync.go:2423 +0x852
github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).DeliverSnapPacket(0xc01bba8960, 0xc09c34ed80, 0x1c4bb60, 0xc10d944900, 0x2d6c7e01, 0xc10d944900)
        github.com/ethereum/go-ethereum@/eth/downloader/downloader.go:2044 +0x347
github.com/ethereum/go-ethereum/eth.(*snapHandler).Handle(0xc0001338c0, 0xc09c34ed80, 0x1c4bb60, 0xc10d944900, 0x3, 0x622732924331ee0b)
        github.com/ethereum/go-ethereum@/eth/handler_snap.go:49 +0x50
github.com/ethereum/go-ethereum/eth/protocols/snap.handleMessage(0x1c66720, 0xc0001338c0, 0xc09c34ed80, 0x0, 0x0)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/handler.go:373 +0x21ea
github.com/ethereum/go-ethereum/eth/protocols/snap.handle(0x1c66720, 0xc0001338c0, 0xc09c34ed80, 0xc02403d700, 0xc0000a2b01)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/handler.go:123 +0x43
github.com/ethereum/go-ethereum/eth/protocols/snap.MakeProtocols.func2.1(0xc09c34ed80, 0xc09c34ed80, 0x0)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/handler.go:103 +0x3d
github.com/ethereum/go-ethereum/eth.(*handler).runSnapExtension(0xc0001338c0, 0xc09c34ed80, 0xc182754980, 0x0, 0x0)
        github.com/ethereum/go-ethereum@/eth/handler.go:374 +0x208
github.com/ethereum/go-ethereum/eth.(*snapHandler).RunPeer(0xc0001338c0, 0xc09c34ed80, 0xc182754980, 0xc06a61aa00, 0xc09c34ed80)
        github.com/ethereum/go-ethereum@/eth/handler_snap.go:33 +0x3f
github.com/ethereum/go-ethereum/eth/protocols/snap.MakeProtocols.func2(0xc0983f28a0, 0x1c4c2e0, 0xc06a61aa00, 0x0, 0xc04dcdc6c0)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/handler.go:102 +0xbb
github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols.func1(0xc0983f28a0, 0xc06a61aa00, 0x1c4c2e0, 0xc06a61aa00)

        github.com/ethereum/go-ethereum@/p2p/peer.go:396 +0xa3
created by github.com/ethereum/go-ethereum/p2p.(*Peer).startProtocols
        github.com/ethereum/go-ethereum@/p2p/peer.go:394 +0x203

Logs doesn't show what was an actual problem:

t=2021-10-26T14:35:46+0300 lvl=info msg="State sync in progress"                 synced=41.48% state="56.41 GiB" accounts=29,722,[email protected] slots=239,049,[email protected] codes=184,[email protected] eta=21m54.622s

t=2021-10-26T14:35:48+0300 lvl=info msg="Imported new block headers"             count=1   elapsed=10.561ms    number=12,105,895 hash=0xe9ef0eae0aa3824d3739dda7c67d559abb8dcdc87e6308d5d057a472822d517c

t=2021-10-26T14:35:49+0300 lvl=eror msg="Snapshot extension registration failed" peer=98998f1a err="peer connected on snap without compatible eth support"

t=2021-10-26T14:35:50+0300 lvl=eror msg="Snapshot extension registration failed" peer=a23dedfc err="peer connected on snap without compatible eth support"

t=2021-10-26T14:35:51+0300 lvl=info msg="Imported new block headers"             count=2   elapsed=19.894ms    number=12,105,897 hash=0x3515ac3fe07c4ca6452a5fe4b37a04659181295fb8ff9a07a3f4a531d594fbed

t=2021-10-26T14:35:54+0300 lvl=info msg="State sync in progress"                 synced=41.50% state="56.51 GiB" accounts=29,737,[email protected] slots=239,510,[email protected] codes=184,[email protected] eta=21m49.253s

t=2021-10-26T14:35:55+0300 lvl=info msg="Imported new block headers"             count=1   elapsed=24.286ms    number=12,105,898 hash=0x68bfe126cf210995989b5738b6c0d4f9b44f19263fdf7efb4ebeef97c07ef31e

t=2021-10-26T14:35:55+0300 lvl=eror msg="Snapshot extension registration failed" peer=5e8011ec err="peer connected on snap without compatible eth support"

**UPD: **
Happend once again. A bit another output:

panic: deletion not supported

goroutine 387018 [running]:
github.com/ethereum/go-ethereum/trie.(*StackTrie).TryUpdate(0xc05f5add40, 0xc11e17a0a0, 0x20, 0x20, 0x29f31d8, 0x0, 0x0, 0xc0d4bcf728, 0xc0a2415e40)
        github.com/ethereum/go-ethereum@/trie/stacktrie.go:194 +0xd0
github.com/ethereum/go-ethereum/trie.(*StackTrie).Update(0xc05f5add40, 0xc11e17a0a0, 0x20, 0x20, 0x29f31d8, 0x0, 0x0)
        github.com/ethereum/go-ethereum@/trie/stacktrie.go:201 +0x77
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).processStorageResponse(0xc00000cd20, 0xc0f6838c00)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/sync.go:1928 +0x3ea
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).Sync(0xc00000cd20, 0xf5dab19f0d4d211d, 0x90df7080e18d0f79, 0x47dca2993be8cfdd, 0xacd7393842fe704f, 0xc0f35fc060, 0x0, 0x0)
        github.com/ethereum/go-ethereum@/eth/protocols/snap/sync.go:653 +0xbfb
github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).run(0xc0707a2280)
        github.com/ethereum/go-ethereum@/eth/downloader/statesync.go:318 +0x87
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).runStateSync
        github.com/ethereum/go-ethereum@/eth/downloader/statesync.go:114 +0x1e9

@webhype
Copy link

webhype commented Oct 26, 2021

Same here, but version 1.1.3 - any ideas?

@grGred
Copy link

grGred commented Oct 26, 2021 •
edited
Loading

Same here, but version 1.1.3 - any ideas?

My node crashed 3 times, and after that while starting i always recieved something like: Detected unclear exit.
After a some moment it stopped crashing. Now it compacts database and continues sync. So I guess you should just monitor it to resume in case of crashing.

@princesscrypto1337
Copy link

I'm running into the same panic after syncing from scratch with snap mode

panic: deletion not supported

goroutine 1531641 [running]:
github.com/ethereum/go-ethereum/trie.(*StackTrie).TryUpdate(0xc053ffcf70, 0xc029140460, 0x20, 0x20, 0x280a478, 0x0, 0x0, 0xc043d8b7e8, 0xc0da2e3e00)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:194 +0xd2
github.com/ethereum/go-ethereum/trie.(*StackTrie).Update(0xc053ffcf70, 0xc029140460, 0x20, 0x20, 0x280a478, 0x0, 0x0)
        github.com/ethereum/go-ethereum/trie/stacktrie.go:201 +0x77
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).processStorageResponse(0xc00000cf00, 0xc04c478600)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:1928 +0x3f4
github.com/ethereum/go-ethereum/eth/protocols/snap.(*Syncer).Sync(0xc00000cf00, 0xcf8fef4722d1f77f, 0x1753f77344dc40fd, 0xaf97cc4260622dc9, 0xd2d6d8a0580567aa, 0xc019de17a0, 0x0, 0x0)
        github.com/ethereum/go-ethereum/eth/protocols/snap/sync.go:653 +0xcae
github.com/ethereum/go-ethereum/eth/downloader.(*stateSync).run(0xc0582ffae0)
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:318 +0x87
created by github.com/ethereum/go-ethereum/eth/downloader.(*Downloader).runStateSync
        github.com/ethereum/go-ethereum/eth/downloader/statesync.go:114 +0x1ef

@b10c77
Copy link

b10c77 commented Oct 31, 2021

Seems to be related to syncing using snap mode, don't get this with the default method but got when using snap

@vrogojin vrogojin mentioned this issue Nov 3, 2021
Closed
@Marijus
Copy link

Marijus commented Nov 9, 2021

facing same issue - any tips?

@vrogojin
Copy link
Author

vrogojin commented Nov 9, 2021

I switched to other physical nvme, and it worked without problem. The first nvme drive itself was fine though. No idea, what geth did not like about it.

@MischaWeerwag
Copy link

MischaWeerwag commented Nov 15, 2021 •
edited
Loading

Same issue. Looks like it can be fixed by increasing the soft/hard limit of open files from 1024 to 65535.

Edit, looks like it isnt completely gone but significantly reduced.

@keefel keefel added the bug Something isn't working label Dec 3, 2021
@ikgo
Copy link

ikgo commented Dec 4, 2021 •
edited
Loading

any status update on the bug.
my node v1.1.5 enter to restart loop. panic after couple of minutes after star syncing.

prune-state exit with message

ERROR[12-04|15:03:04.063] Failed to prune state                    err="snapshot not old enough yet: need 128 more blocks"
snapshot not old enough yet: need 128 more blocks

is a version 1.1.6 fix this bug?

@seaify
Copy link

seaify commented Dec 7, 2021

when can it be fixed? can't sync

@keefel keefel mentioned this issue Dec 16, 2021
Merged
@keefel
Copy link
Contributor

keefel commented Dec 16, 2021

Fixed by #667, please wait for next release.

@keefel keefel closed this as completed Dec 31, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@b10c77 @Marijus @seaify @MischaWeerwag @webhype @vrogojin @ikgo @grGred @princesscrypto1337 @keefel