Skip to content

Delegate Q&A

Yutong Pei edited this page Oct 17, 2020 · 11 revisions

Q: Node is not producing block

First please check if node height is still increasing or not, if so please jump to here, if not please jump to here.

Q: Node is not producing block, but node height is increasing

First make sure, you are selected in current epoch as block producers. You can check through ioctl node delegate

If you are elected as block producers, please check if your node is keeping up to current chain tip height. If not you can download latest snapshot to speed up the catch up process.

If you are elected and your node is keeping up to chain tip height, probably your operator private key you set in config is not the same one you set in member.iotex.io delegate configuration.

If you are using our upgrader, please run the upgrader command again and follow the instructions to enter your correct operator private key.

If you are not using our upgrader, you should set it in config.yaml and restart your node:

chain:
  producerPrivKey: <SET YOUR PRIVATE KEY HERE>
  # If you are a delegate, make sure producerPrivKey is the key for the operator address you have registered.

If you don't have your operator key any more, you can set it to a new one in member.iotex.io with your owner key under delegate configuration. You can generate a new key pair using ioctl command: ioctl account create

If you lost your owner key, there is nothing your can do, so please make sure you keep your owner key safe.

Q: Node height is not increasing

Please make sure you already upgraded the node into latest version with correct configuration first.

If everything is up to date as iotex-bootstrap, please check your log in this case. You can dump log from docker by using command:

docker logs [container_name] --tail [num of line, e.g. 3000] > output.log
# for example, sudo docker logs iotex --tail 3000 > output.log

Please check log against following error digests, if your issues still cannot be addressed, please contact us directly and send us your log.

Q: What if I see log like this when node height is not increasing (1)

Failed to new config.
{error 25 0  failed to unmarshal YAML config to struct: yaml: unmarshal errors:...}

Your config.yaml or genesis.yaml is not formatted correctly. Please check these files if you have a typo or an indent error. Easy way is download again from iotex-bootstrap and change private key and external ip to yours, then restart node.

Q: What if I see log like this when node height is not increasing (2)

Failed to new config.
{error 25 0 error when constructing a xxx in yaml: error applying options: read /etc/iotex/xxx.yaml: ...}
open /var/data/XXXX.db: no such file or directory: DB I/O operation error

Make sure you set your env var $IOTEX_HOME correctly.

Make sure your mount all the files and paths correctly when you run docker($IOTEX_HOME/data, $IOTEX_HOME/etc/config.yaml, $IOTEX_HOME/etc/genesis.yaml).

In data folder, you should have these db files: candidate.index.db chain.db consensus.db index.db poll.db trie.db. If you miss any of them, please download the latest snapshot.

If you don't use upgrader or docker command we provided in iotex-bootstrap, please make sure you update all db paths in config.yaml to the right location if you don't put db files under /var/data/.

Q: What if I see log like this when node height is not increasing (3)

"Finished indexing blocks up to"

This is because node is blocked on indexing blocks. You can either wait the indexing catch up to your node's height or download the latest snapshot with index(download here) to start your node which could be faster.

Q: What if I see log like this when node height is not increasing (4)

401 Unauthorized account disabled\n\nfailed to execute callback with any client\ngithub.com/iotexproject/iotex-election/carrier.(*EthClientPool)
"project ID request rate exceeded"
"see":"https://infura.io/docs/ethereum/json-rpc/ratelimits"

These two errors shouldn't happen if your node block height is above 5165641. Probably you are trying to sync from scratch(0 height) or an earlier height. Before height 5165641, IoTeX mainnet staking and election data is on Ethereum. And the error msg above is because infura API got ratelimit. You have 3 options to solve this error:

  1. (Recommended) Use our latest snapshot(with-index or without-index) to catch up so you don't need to get data from Ethereum.
  2. (Recommended) Download the complete legacy delegate election data so you can sync IoTeX chain from scratch and also don't need to get data from Ethereum:
curl -L https://storage.googleapis.com/blockchain-golden/poll.mainnet.tar.gz > $IOTEX_HOME/poll.tar.gz; tar -xzf $IOTEX_HOME/poll.tar.gz --directory $IOTEX_HOME/data
  1. Change to use your own infura key with Ethereum archive mode supported or change the API endpoint to an Ethereum archive node which you can access with:
  committee:
    gravityChainAPIs:
      # please change the infura key to your key (e.g., https://mainnet.infura.io/v3/YOUR_KEY)

Q: What if I see log like this when node height is not increasing (5)

"Failed to commit the block."

Please make sure your node is running the latest version and use correct configurations. If in that case, you still have this issue, please contact us directly.

Q: What if I see log like this? (1)

"msg":"Error when serving performance profiling data.", "error":"http: Server closed"}
"msg":"Error when subscribing a broadcast message.", "error":"subscription cancelled by calling sub.Cancel()"

This is a Normal log message you shall see, when node stop/restart. There is nothing special about this log itself. It is more important what cause the node to stop/restart. It could be your maintenance engineer restart it. So please check the logs Before this log to get better understanding

Q: Why my delegate shows unhealthy on member.iotex.io?

Please make sure you set your health endpoint correctly in member.iotex.io delegate configuration. And wait couple minutes to make sure website catch up the change.