Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot restart syncing archive node #1694

Closed
AltOrder opened this issue Jul 14, 2023 · 1 comment
Closed

Cannot restart syncing archive node #1694

AltOrder opened this issue Jul 14, 2023 · 1 comment

Comments

@AltOrder
Copy link

Starting and stopping a syncing archive node fails for Kusama and Polkadot. This happens both in Full and Fast mode.

  • Docker image
soramitsu/kagome                       latest    f896e533d8c6   29 hours ago   464MB
  • Reproduce with Full mode on Kusama:
docker run -d -v /root:/kagome --name kagome soramitsu/kagome:latest kagome --chain kusama --base-path kusama-node-1 --sync Full --state-pruning archive
docker restart kagome

This hangs kagome and needs to be killed manually.

docker logs kagome -f
...
...
23.07.14 19:30:28.992889  kagome           Info      BlockExecutor  Imported block #484 (0x8e38…b0c1) within 28 ms. (lag 1324 days)
23.07.14 19:30:29.022523  kagome           Info      BlockStorage  Added block #485 (0x92f7…62ac) as child of #484 (0x8e38…b0c1)
23.07.14 19:30:29.022667  kagome           Info      BlockExecutor  Imported block #485 (0x92f7…62ac) within 28 ms. (lag 1324 days)
Fatal: callImport: unknown import: env.��]^AV
23.07.14 19:30:47.467629  kagome           Info      AppConfiguration  Soramitsu Kagome started. Version: 0.9.0-70
23.07.14 19:30:47.468121  kagome           Info      AppConfiguration  Automatically added IPv6 listen address /ip6/::/tcp/30363
23.07.14 19:30:47.468154  kagome           Info      AppConfiguration  Automatically added IPv4 listen address /ip4/0.0.0.0/tcp/30363
23.07.14 19:30:47.468341  kagome           Info      AppConfiguration  Listen addresses are set. The p2p port value would be ignored then.
23.07.14 19:30:47.526633  kagome           Warning   chain_spec  Field 'chainType' was not specified in the chain spec. 'Live' by default.
23.07.14 19:30:47.526691  kagome           Warning   chain_spec  A non-empty set of 'badBlocks' encountered! They might not be taken into account!
23.07.14 19:30:47.836981  kagome           Warning   Injector  Can not obtain a libp2p keypair from crypto storage. A unique one will be generated
23.07.14 19:30:47.847528  kagome           Info      BlockTree  Highest block: #485 (0x92f7…62ac), Last finalized: #0 (0xb0a8…dafe)
23.07.14 19:30:47.856044  kagome           Warning   BlockTree  Can't add existing non-finalized block #0 (0xb0a8…dafe) to block tree: block, which should have been inserted, already exists in the tree
23.07.14 19:30:47.952965  kagome           Warning   PrunerStateRecovery  Running pruner on a non-empty non-pruned storage may lead to skipping some stored states.
  • For Fast mode, kagome segfaults on restart (with clean database):
...
...
23.07.14 19:34:12.666719  kagome           Info      BlockTree  Finalized block #1024 (0x3f17…fd7f)
23.07.14 19:34:13.680902  kagome           Info      BlockTree  Finalized block #1536 (0x55d1…fb04)
23.07.14 19:34:14.556288  kagome           Info      AppConfiguration  Soramitsu Kagome started. Version: 0.9.0-70
23.07.14 19:34:14.556729  kagome           Info      AppConfiguration  Automatically added IPv6 listen address /ip6/::/tcp/30363
23.07.14 19:34:14.556764  kagome           Info      AppConfiguration  Automatically added IPv4 listen address /ip4/0.0.0.0/tcp/30363
23.07.14 19:34:14.556998  kagome           Info      AppConfiguration  Listen addresses are set. The p2p port value would be ignored then.
23.07.14 19:34:14.615219  kagome           Warning   chain_spec  Field 'chainType' was not specified in the chain spec. 'Live' by default.
23.07.14 19:34:14.615279  kagome           Warning   chain_spec  A non-empty set of 'badBlocks' encountered! They might not be taken into account!
23.07.14 19:34:14.885920  kagome           Warning   Injector  Can not obtain a libp2p keypair from crypto storage. A unique one will be generated
23.07.14 19:34:14.890650  kagome           Info      BlockTree  Highest block: #1665 (0xcddc…e084), Last finalized: #1536 (0x55d1…fb04)
23.07.14 19:34:14.893108  kagome           Warning   BlockTree  Can't add existing non-finalized block #1536 (0x55d1…fb04) to block tree: block, which should have been inserted, already exists in the tree
23.07.14 19:34:14.976922  kagome           Warning   PrunerStateRecovery  Running pruner on a non-empty non-pruned storage may lead to skipping some stored states.
23.07.14 19:34:14.978201  kagome           Info      Application  Start as node version '0.9.0-70' named as 'ebaefdc1-eccb-40d0-910d-4e3bb0b9e3f7' with PID 1
23.07.14 19:34:14.978240  kagome           Info      Application  Chain path is "kusama-node-1/chains/ksmcc3", storage backend is RocksDB
terminate called after throwing an instance of 'boost::wrapexcept<std::system_error>'
  what():  the block must contain at least BABE header and seal digests
Stack trace (most recent call last):
#19   Object "kagome", at 0x555e349c32c0, in
#18   Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d5008784, in __libc_start_main
#17   Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d50086c9, in
#16   Object "kagome", at 0x555e348c79af, in
#15   Object "kagome", at 0x555e349cce50, in
#14   Object "kagome", at 0x555e4072faa8, in
#13   Object "kagome", at 0x555e4072fe8a, in
#12   Object "kagome", at 0x555e3d87bcf0, in
#11   Object "kagome", at 0x555e347cf2ea, in
#10   Object "kagome", at 0x555e349c68de, in
#9    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f57d53522b7, in __cxa_throw
#8    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f57d5352064, in std::terminate()
#7    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f57d5351ff9, in
#6    Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f57d53429da, in
#5    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d50074b1, in abort
#4    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d501d471, in gsignal
#3    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d506b0fb, in
#2    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f57d501d50f, in
#1    Object "kagome", at 0x555e349c9fee, in
#0    Object "kagome", at 0x555e349c9046, in
Aborted (Signal sent by tkill() 1 0)
watf? exit
@Harrm
Copy link
Contributor

Harrm commented Jul 18, 2023

This happens due to some problems with the new trie pruner, when #1702 is merged you will be able to run the node without it, while it's still unstable.

@kamilsa kamilsa closed this as completed Nov 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants