You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The node was up for 2 days and running as a fail over. Being a 'fail over' node means that the chain data was, at some time, being used by a validator node, then the same chain data was used for a full node. We use fullnode's as failover so there is a complete synced datadir ready.
Currently the node goes into a bootloop with the error:
{"log":"INFO [09-11|08:04:31.348] Starting peer-to-peer node instance=celo/v1.0.1-stable/linux-amd64/go1.13.14\n","stream":"stderr","time":"2020-09-11T08:04:31.348293711Z"}
{"log":"INFO [09-11|08:04:31.348] Allocated trie memory caches clean=256.00MiB dirty=256.00MiB\n","stream":"stderr","time":"2020-09-11T08:04:31.348311647Z"}
{"log":"INFO [09-11|08:04:31.348] Allocated cache and file handles database=/root/.celo/celo/chaindata cache=768.00MiB handles=524288\n","stream":"stderr","time":"2020-09-11T08:04:31.348350879Z"}
{"log":"INFO [09-11|08:04:31.371] Opened ancient database database=/root/.celo/celo/chaindata/ancient\n","stream":"stderr","time":"2020-09-11T08:04:31.37165527Z"}
{"log":"Fatal: Error starting protocol stack: gap (#2151176) in the chain between ancients and leveldb\n","stream":"stderr","time":"2020-09-11T08:04:31.425256486Z"}
{"log":"Fatal: Error starting protocol stack: gap (#2151176) in the chain between ancients and leveldb\n","stream":"stdout","time":"2020-09-11T08:04:31.425282025Z"}
Steps to Reproduce Behavior
Restart the fullnode container with the same datadir
Logs
{"log":"INFO [09-11|05:48:28.253] Ethereum peer connected id=dc7019cea4f6fa8d conn=inbound name=celo/v1.0.1-stable/linux-amd64/go1.13.12\n","stream":"stderr","time":"2020-09-11T05:48:28.253949337Z"}
{"log":"INFO [09-11|05:48:28.269] Ethereum peer connected id=9cf07fc521286dee conn=inbound name=celo/v1.0.1-stable/linux-amd64/go1.13.12\n","stream":"stderr","time":"2020-09-11T05:48:28.26922345Z"}
{"log":"INFO [09-11|05:48:28.508] Imported new chain segment blocks=1 txs=0 mgas=0.000 elapsed=12.485ms mgasps=0.000 number=2445739 hash=3a27dd…ac6e81 dirty=7.93MiB\n","stream":"stderr","time":"2020-09-11T05:48:28.50839709Z"}
{"log":"INFO [09-11|05:48:28.657] Ethereum peer connected id=7732e386b1db651b conn=inbound name=celo/v1.0.0-stable/linux-amd64/go1.13.10\n","stream":"stderr","time":"2020-09-11T05:48:28.65792297Z"}
�������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������{"log":"stty: standard input: Not a tty\n","stream":"stderr","time":"2020-09-11T05:49:13.70023394Z"}
{"log":"INFO [09-11|05:49:15.005] Maximum peer count ETH=100 LES=1000 total=1100\n","stream":"stderr","time":"2020-09-11T05:49:15.007121456Z"}
{"log":"INFO [09-11|05:49:15.007] Smartcard socket not found, disabling err=\"stat /run/pcscd/pcscd.comm: no such file or directory\"\n","stream":"stderr","time":"2020-09-11T05:49:15.008645072Z"}
{"log":"WARN [09-11|05:49:15.025] Found deprecated node list file /root/.celo/celo/static-nodes.json, please use the TOML config file instead. \n","stream":"stderr","time":"2020-09-11T05:49:15.027015355Z"}
{"log":"INFO [09-11|05:49:15.028] Starting peer-to-peer node instance=celo/v1.0.1-stable/linux-amd64/go1.13.14\n","stream":"stderr","time":"2020-09-11T05:49:15.028380235Z"}
{"log":"INFO [09-11|05:49:15.028] Allocated trie memory caches clean=256.00MiB dirty=256.00MiB\n","stream":"stderr","time":"2020-09-11T05:49:15.031999621Z"}
{"log":"INFO [09-11|05:49:15.031] Allocated cache and file handles database=/root/.celo/celo/chaindata cache=768.00MiB handles=524288\n","stream":"stderr","time":"2020-09-11T05:49:15.034227288Z"}
{"log":"INFO [09-11|05:49:15.528] Opened ancient database database=/root/.celo/celo/chaindata/ancient\n","stream":"stderr","time":"2020-09-11T05:49:15.528321983Z"}
{"log":"INFO [09-11|05:49:15.596] Initialised chain configuration config=\"{ChainID: 42220 Homestead: 0 DAO: \u003cnil\u003e DAOSupport: false EIP150: 0 EIP155: 0 EIP158: 0 Byzantium: 0 Constantinople: 0 Petersburg: 0 Istanbul: 0 Engine: istanbul}\"\n","stream":"stderr","time":"2020-09-11T05:49:15.597011668Z"}
{"log":"INFO [09-11|05:49:15.597] Open roundstate db func=newRoundStateDB type=roundStateDB rsdb_path=/root/.celo/celo/roundstates\n","stream":"stderr","time":"2020-09-11T05:49:15.5989893Z"}
{"log":"INFO [09-11|05:49:15.737] Initialising Ethereum protocol versions=\"[65 64]\" network=42220 dbversion=7\n","stream":"stderr","time":"2020-09-11T05:49:15.737593374Z"}
{"log":"WARN [09-11|05:49:15.744] Head state missing, repairing chain number=2445739 hash=3a27dd…ac6e81\n","stream":"stderr","time":"2020-09-11T05:49:15.744723862Z"}
{"log":"INFO [09-11|05:50:01.408] Rewound blockchain to past state number=2151175 hash=6be2f4…7fc9c5\n","stream":"stderr","time":"2020-09-11T05:50:01.408890712Z"}
{"log":"INFO [09-11|05:50:01.410] Loaded most recent local header number=2445739 hash=3a27dd…ac6e81 td=2445740 age=1m33s\n","stream":"stderr","time":"2020-09-11T05:50:01.410626279Z"}
{"log":"INFO [09-11|05:50:01.410] Loaded most recent local full block number=2151175 hash=6be2f4…7fc9c5 td=2151176 age=2w3d1h\n","stream":"stderr","time":"2020-09-11T05:50:01.411110727Z"}
{"log":"INFO [09-11|05:50:01.410] Loaded most recent local fast block number=2445739 hash=3a27dd…ac6e81 td=2445740 age=1m33s\n","stream":"stderr","time":"2020-09-11T05:50:01.411117636Z"}
�����������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������������{"log":"stty: standard input: Not a tty\n","stream":"stderr","time":"2020-09-11T05:51:05.588676597Z"}
{"log":"INFO [09-11|05:51:06.795] Maximum peer count ETH=100 LES=1000 total=1100\n","stream":"stderr","time":"2020-09-11T05:51:06.797511879Z"}
{"log":"INFO [09-11|05:51:06.798] Smartcard socket not found, disabling err=\"stat /run/pcscd/pcscd.comm: no such file or directory\"\n","stream":"stderr","time":"2020-09-11T05:51:06.799240126Z"}
{"log":"WARN [09-11|05:51:06.807] Found deprecated node list file /root/.celo/celo/static-nodes.json, please use the TOML config file instead. \n","stream":"stderr","time":"2020-09-11T05:51:06.80919953Z"}
{"log":"INFO [09-11|05:51:06.809] Starting peer-to-peer node instance=celo/v1.0.1-stable/linux-amd64/go1.13.14\n","stream":"stderr","time":"2020-09-11T05:51:06.81010782Z"}
{"log":"INFO [09-11|05:51:06.810] Allocated trie memory caches clean=256.00MiB dirty=256.00MiB\n","stream":"stderr","time":"2020-09-11T05:51:06.812451789Z"}
{"log":"INFO [09-11|05:51:06.812] Allocated cache and file handles database=/root/.celo/celo/chaindata cache=768.00MiB handles=524288\n","stream":"stderr","time":"2020-09-11T05:51:06.813468627Z"}
{"log":"INFO [09-11|05:51:08.265] Opened ancient database database=/root/.celo/celo/chaindata/ancient\n","stream":"stderr","time":"2020-09-11T05:51:08.265942346Z"}
{"log":"Fatal: Error starting protocol stack: gap (#2151176) in the chain between ancients and leveldb\n","stream":"stdout","time":"2020-09-11T05:51:08.342287214Z"}
{"log":"Fatal: Error starting protocol stack: gap (#2151176) in the chain between ancients and leveldb\n","stream":"stderr","time":"2020-09-11T05:51:08.342335192Z"}
{"log":"stty: standard input: Not a tty\n","stream":"stderr","time":"2020-09-11T05:51:09.933566549Z"}
{"log":"INFO [09-11|05:51:11.109] Maximum peer count ETH=100 LES=1000 total=1100\n","stream":"stderr","time":"2020-09-11T05:51:11.109444316Z"}
{"log":"INFO [09-11|05:51:11.109] Smartcard socket not found, disabling err=\"stat /run/pcscd/pcscd.comm: no such file or directory\"\n","stream":"stderr","time":"2020-09-11T05:51:11.109563543Z"}
{"log":"WARN [09-11|05:51:11.110] Found deprecated node list file /root/.celo/celo/static-nodes.json, please use the TOML config file instead. \n","stream":"stderr","time":"2020-09-11T05:51:11.110930013Z"}
{"log":"INFO [09-11|05:51:11.111] Starting peer-to-peer node instance=celo/v1.0.1-stable/linux-amd64/go1.13.14\n","stream":"stderr","time":"2020-09-11T05:51:11.111476102Z"}
{"log":"INFO [09-11|05:51:11.111] Allocated trie memory caches clean=256.00MiB dirty=256.00MiB\n","stream":"stderr","time":"2020-09-11T05:51:11.111590996Z"}
{"log":"INFO [09-11|05:51:11.111] Allocated cache and file handles database=/root/.celo/celo/chaindata cache=768.00MiB handles=524288\n","stream":"stderr","time":"2020-09-11T05:51:11.111924759Z"}
{"log":"INFO [09-11|05:51:11.166] Opened ancient database database=/root/.celo/celo/chaindata/ancient\n","stream":"stderr","time":"2020-09-11T05:51:11.166801885Z"}
{"log":"Fatal: Error starting protocol stack: gap (#2151176) in the chain between ancients and leveldb\n","stream":"stdout","time":"2020-09-11T05:51:11.283987917Z"}
{"log":"Fatal: Error starting protocol stack: gap (#2151176) in the chain between ancients and leveldb\n","stream":"stderr","time":"2020-09-11T05:51:11.28408006Z"}
{"log":"stty: standard input: Not a tty\n","stream":"stderr","time":"2020-09-11T05:51:12.318003712Z"}
System information
celo v1.0.1-stable
The text was updated successfully, but these errors were encountered:
Expected Behavior
fullnode would continue syncing with the chain.
The node was up for 2 days and running as a fail over. Being a 'fail over' node means that the chain data was, at some time, being used by a validator node, then the same chain data was used for a full node. We use fullnode's as failover so there is a complete synced datadir ready.
Although the fullnode was running for 2 days with no problem. This could also be an upstream issue with
geth
, see discord: https://discordapp.com/channels/600834479145353243/697229909545713775/754011794791596065 and ethereum/go-ethereum#21409Current Behavior
Currently the node goes into a bootloop with the error:
Steps to Reproduce Behavior
Restart the fullnode container with the same datadir
Logs
System information
celo v1.0.1-stable
The text was updated successfully, but these errors were encountered: