Skip to content
This repository has been archived by the owner on Jun 11, 2024. It is now read-only.

Failed to generate blocks in v4.0.0-beta.1 #8587

Closed
Tracked by #7226
punkrock opened this issue Jun 8, 2023 · 16 comments
Closed
Tracked by #7226

Failed to generate blocks in v4.0.0-beta.1 #8587

punkrock opened this issue Jun 8, 2023 · 16 comments
Assignees
@shuse2
Labels
framework type: bug
Milestone
Sprint 100

Comments

@punkrock
Copy link

punkrock commented Jun 8, 2023 •
edited
Loading

In v4.0.0-beta.1 the node stopped to generate blocks with the following error. The chain on the node kept moving but stopped to generate blocks.

Error (no other errors in logs):

2023-06-08T02:09:33.225Z ERROR betanet engine 332042 [err=Execution context is already initialized with 6e0dd8df5d840a67ff8bfc7f291e8de2bf65ad1f86399a4d05ac7c8e9c41e643] Failed to generate a block
/home/lisk/lisk-core/bin/lisk-core: line 54: 332042 Killed                  "$NODE" "$DIR/run" "$@"
@sameersubudhi
Copy link

@punkrock What is the environment (OS) you're running the node in?

@punkrock
Copy link
Author

punkrock commented Jun 8, 2023

@sameersubudhi Ubuntu 22.04.2 LTS

@sameersubudhi
Copy link

Did you happen to run the same generator (w/ same keys) on another node simultaneously?

@punkrock
Copy link
Author

punkrock commented Jun 8, 2023

@sameersubudhi No, this is my only node and I did nothing. The error just showed up and my node stopped generating blocks but the chain kept moving like nothing happened.

@punkrock
Copy link
Author

punkrock commented Jun 9, 2023 •
edited
Loading

Today it happened again. Here the full logs (notice the time difference between logs!):

2023-06-09T01:37:27.287Z WARN betanet engine 505075 [error=Apply penalty and restart synchronization mechanism with reason: Peer didn't return any block after requesting blocks reason=Peer didn't return any block after requesting blocks] Applying penalty and restarting synchronization
2023-06-09T01:37:35.249Z ERROR betanet engine 505075 [peerId=216.238.66.142:7667 err=List of connected peers is empty] Fail to handle received block
2023-06-09T01:42:59.780Z ERROR betanet engine 505075 [err=Error: Contradicting headers for the block with id: 96d20e790af93ac6f4d98bfe8b25690ac76413ffbc21a2a3c1b10cca4144f42c] Fail to execute block.
2023-06-09T01:44:11.624Z ERROR betanet engine 505075 [err=Event response for 'rpc-request' timed out] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-09T01:45:26.249Z WARN betanet engine 505075 [error=Apply penalty and restart synchronization mechanism with reason: Peer didn't return any block after requesting blocks reason=Peer didn't return any block after requesting blocks] Applying penalty and restarting synchronization
2023-06-09T01:45:31.537Z ERROR betanet engine 505075 [peerId=45.32.203.143:7667 err=List of connected peers is empty] Fail to handle received block
2023-06-09T01:47:41.376Z ERROR betanet engine 505075 [err=Event response for 'rpc-request' timed out] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-09T01:47:54.078Z ERROR betanet engine 505075 [peerId=45.32.203.143:7667 err=Event response for 'rpc-request' timed out] Fail to handle received block
2023-06-09T01:48:06.801Z ERROR betanet engine 505075 [peerId=45.32.203.143:7667 err=Event response for 'rpc-request' timed out] Fail to handle received block
2023-06-09T01:48:38.674Z ERROR betanet engine 505075 [peerId=45.32.203.143:7667 err=Event response for 'rpc-request' timed out] Fail to handle received block
/home/lisk/lisk-core/bin/lisk-core: line 54: 505075 Killed                  "$NODE" "$DIR/run" "$@"
2023-06-09T08:45:52.162Z ERROR betanet engine 516984 [err=Event response for 'rpc-request' timed out] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-09T10:40:36.531Z ERROR betanet engine 516984 [err=Event response for 'rpc-request' timed out] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-09T10:40:36.751Z ERROR betanet engine 516984 [err=Event response for 'rpc-request' timed out] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-09T10:40:36.892Z ERROR betanet engine 516984 [err=Event response for 'rpc-request' timed out] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-09T10:40:37.056Z ERROR betanet engine 516984 [err=Event response for 'rpc-request' timed out] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-09T10:52:57.382Z ERROR betanet engine 516984 [err=Event response for 'rpc-request' timed out] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-09T10:55:08.846Z ERROR betanet engine 516984 [peerId=20.48.103.139:7667 err=Event response for 'rpc-request' timed out] Fail to handle received block
2023-06-09T11:20:33.118Z ERROR betanet engine 516984 [err=Event response for 'rpc-request' timed out] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
/home/lisk/lisk-core/bin/lisk-core: line 54: 516984 Killed                  "$NODE" "$DIR/run" "$@"

@sameersubudhi
Copy link

@punkrock Could you please run the node with trace logs and share them when this happens again?

@ManuGowda
Copy link
Contributor

Moving this issue to SDK for further resolution.

@punkrock
Copy link
Author

@sameersubudhi

I am failing to generate blocks again. Here the error logs (trace):

2023-06-12T03:42:11.199Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:11.355Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:11.535Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.090Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.815Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.822Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.822Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.822Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:12.823Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:42:30.781Z ERROR betanet engine 573312 [peerId=45.32.203.143:7667 err=Apply penalty and abort synchronization mechanism with reason: Peer didn't return a common block trace=ApplyPenaltyAndAbortError: Apply penalty and abort synchronization mechanism with reason: Peer didn't return a common block
    at new SynchronizerError (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/errors.js:5:1)
    at new ApplyPenaltyAndAbortError (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/errors.js:35:9)
    at FastChainSwitchingMechanism._queryBlocks (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/fast_chain_switching_mechanism.js:61:19)
    at FastChainSwitchingMechanism.run (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/fast_chain_switching_mechanism.js:15:35)
    at runMicrotasks (<anonymous>)
    at runNextTicks (node:internal/process/task_queues:61:5)
    at processTimers (node:internal/timers:499:9)
    at async /home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/synchronizer.js:47:13
    at async Mutex.runExclusive (/home/lisk/lisk-core/node_modules/@liskhq/lisk-utils/dist-node/job_handlers/mutex.js:23:20)
    at async Synchronizer.run (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/synchronizer.js:33:9)] Fail to handle received block
2023-06-12T03:43:18.448Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:18.771Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:19.134Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:19.258Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:19.377Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:19.426Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T03:43:19.606Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T08:34:22.246Z ERROR betanet engine 573312 [peerId=80.211.194.139:7667 err=List of connected peers is empty trace=Error: List of connected peers is empty
    at BlockSynchronizationMechanism._computeBestPeer (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/block_synchronization_mechanism.js:231:19)
    at BlockSynchronizationMechanism.run (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/block_synchronization_mechanism.js:28:31)
    at /home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/synchronizer.js:47:34
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async Mutex.runExclusive (/home/lisk/lisk-core/node_modules/@liskhq/lisk-utils/dist-node/job_handlers/mutex.js:23:20)
    at async Synchronizer.run (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/synchronizer.js:33:9)
    at async Consensus._sync (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:584:13)
    at async /home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:310:17
    at async Mutex.runExclusive (/home/lisk/lisk-core/node_modules/@liskhq/lisk-utils/dist-node/job_handlers/mutex.js:23:20)] Fail to handle received block
2023-06-12T09:04:41.609Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
2023-06-12T09:07:41.608Z ERROR betanet engine 573312 [err=Event response for 'rpc-request' timed out trace=TimeoutError: Event response for 'rpc-request' timed out
    at SCClientSocket._handleEventAckTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:678:17)
    at Timeout._onTimeout (/home/lisk/lisk-core/node_modules/socketcluster-client/lib/scclientsocket.js:708:10)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7)] EVENT_FAILED_TO_FETCH_PEER_INFO: Failed to fetch peer info
/home/lisk/lisk-core/bin/lisk-core: line 54: 573312 Killed                  "$NODE" "$DIR/run" "$@"
2023-06-12T13:31:50.548Z ERROR betanet engine 581650 [err=Incoming transaction fee is not sufficient to replace existing transaction trace=TransactionPoolError: Incoming transaction fee is not sufficient to replace existing transaction
    at TransactionPool.add (/home/lisk/lisk-core/node_modules/@liskhq/lisk-transaction-pool/dist-node/transaction_pool.js:127:24)
    at async NetworkEndpoint._receiveTransaction (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:162:27)
    at async NetworkEndpoint.handleEventPostTransactionsAnnouncement (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:132:21)] Failed to add transaction to pool.
2023-06-12T13:31:51.151Z ERROR betanet engine 581650 [err=Incoming transaction fee is not sufficient to replace existing transaction trace=TransactionPoolError: Incoming transaction fee is not sufficient to replace existing transaction
    at TransactionPool.add (/home/lisk/lisk-core/node_modules/@liskhq/lisk-transaction-pool/dist-node/transaction_pool.js:127:24)
    at async NetworkEndpoint._receiveTransaction (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:162:27)
    at async NetworkEndpoint.handleEventPostTransactionsAnnouncement (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:132:21)] Failed to add transaction to pool.
2023-06-12T13:34:23.619Z ERROR betanet engine 581650 [err=Incoming transaction fee is not sufficient to replace existing transaction trace=TransactionPoolError: Incoming transaction fee is not sufficient to replace existing transaction
    at TransactionPool.add (/home/lisk/lisk-core/node_modules/@liskhq/lisk-transaction-pool/dist-node/transaction_pool.js:127:24)
    at async NetworkEndpoint._receiveTransaction (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:162:27)
    at async NetworkEndpoint.handleEventPostTransactionsAnnouncement (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:132:21)] Failed to add transaction to pool.
2023-06-12T13:35:38.604Z ERROR betanet engine 581650 [err=Incoming transaction fee is not sufficient to replace existing transaction trace=TransactionPoolError: Incoming transaction fee is not sufficient to replace existing transaction
    at TransactionPool.add (/home/lisk/lisk-core/node_modules/@liskhq/lisk-transaction-pool/dist-node/transaction_pool.js:127:24)
    at async NetworkEndpoint._receiveTransaction (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:162:27)
    at async NetworkEndpoint.handleEventPostTransactionsAnnouncement (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:132:21)] Failed to add transaction to pool.
/home/lisk/lisk-core/bin/lisk-core: line 54: 581650 Killed                  "$NODE" "$DIR/run" "$@"
/home/lisk/lisk-core/bin/lisk-core: line 54: 586594 Killed                  "$NODE" "$DIR/run" "$@"

@sameersubudhi sameersubudhi transferred this issue from another repository Jun 13, 2023
@punkrock
Copy link
Author

And again:

2023-06-13T09:09:36.149Z ERROR betanet engine 593418 [err=Incoming transaction fee is not sufficient to replace existing transaction trace=TransactionPoolError: Incoming transaction fee is not sufficient to replace existing transaction
    at TransactionPool.add (/home/lisk/lisk-core/node_modules/@liskhq/lisk-transaction-pool/dist-node/transaction_pool.js:127:24)
    at async NetworkEndpoint._receiveTransaction (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:162:27)
    at async NetworkEndpoint.handleEventPostTransactionsAnnouncement (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:132:21)] Failed to add transaction to pool.
2023-06-13T09:12:25.898Z ERROR betanet engine 593418 [err=Incoming transaction fee is not sufficient to replace existing transaction trace=TransactionPoolError: Incoming transaction fee is not sufficient to replace existing transaction
    at TransactionPool.add (/home/lisk/lisk-core/node_modules/@liskhq/lisk-transaction-pool/dist-node/transaction_pool.js:127:24)
    at async NetworkEndpoint._receiveTransaction (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:162:27)
    at async NetworkEndpoint.handleEventPostTransactionsAnnouncement (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/network_endpoint.js:132:21)] Failed to add transaction to pool.
2023-06-13T09:15:31.260Z WARN betanet engine 593418 [data=0ae3020ac202080210a1e7a0a40618ddfe0d2220769236f1b81dcc0dc7de4e6cc7e7e28210cca86ef52e12c4c608366298d201032a146860b677c20acc03ce315c3550adb92b3b955d6b3220e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b8553a20921a90131d16dc92515845e6347708346581975c56fed57b257a58880f8e7b3142202400204c34c0167248fe9696cb0fb9d7f999b93d6bb4757741e2587cc84d28be4a2070ffb966caba8f5b3a8a258121073816fb1da48de7b1922229bd9f670d6335dc50cefe0d58c8fe0d60016a205867c8993cecbbccc53e74ab5561119cf5a6996014cc24c44fa4f8a38bfa128f7206080012001a007a40d0080a1a026832f86bccbf79a988beff7042f90c08e13a956633f9a0ca7eb7c3c7f36ecca7cdbed82f937f497d2e3c16e0a8049e38ca17b99b260aa97adaeb091a1c0a0672616e646f6d12120a1042295edf8f68bccce248eb1b5b171d80 err=Error: Execution context is already initialized with efd4c6440e759cb1e6ac596f54380464814915919928192b74b66f4dbab03af3 trace=ApplyPenaltyError: Error: Execution context is already initialized with efd4c6440e759cb1e6ac596f54380464814915919928192b74b66f4dbab03af3
    at new ApplyPenaltyError (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/errors.js:74:9)
    at Consensus._executeValidated (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:381:19)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:359:13
    at async Mutex.runExclusive (/home/lisk/lisk-core/node_modules/@liskhq/lisk-utils/dist-node/job_handlers/mutex.js:23:20)
    at async Consensus._execute (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:252:9)
    at async Consensus.onBlockReceive (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:192:13)] Received post block broadcast request with invalid block. Applying a penalty to the peer
2023-06-13T09:15:31.267Z ERROR betanet engine 593418 [peerId=80.211.194.139:7667] EVENT_MESSAGE_RECEIVED: Peer has been banned temporarily
2023-06-13T09:15:31.267Z ERROR betanet engine 593418 [peerId=80.211.194.139:7667 err=Error: Execution context is already initialized with efd4c6440e759cb1e6ac596f54380464814915919928192b74b66f4dbab03af3 trace=ApplyPenaltyError: Error: Execution context is already initialized with efd4c6440e759cb1e6ac596f54380464814915919928192b74b66f4dbab03af3
    at new ApplyPenaltyError (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/errors.js:74:9)
    at Consensus._executeValidated (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:381:19)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:359:13
    at async Mutex.runExclusive (/home/lisk/lisk-core/node_modules/@liskhq/lisk-utils/dist-node/job_handlers/mutex.js:23:20)
    at async Consensus._execute (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:252:9)
    at async Consensus.onBlockReceive (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:192:13)] Fail to handle received block
2023-06-13T09:15:31.321Z ERROR betanet engine 593418 [err=Cannot read properties of undefined (reading 'stateStore') trace=TypeError: Cannot read properties of undefined (reading 'stateStore')
    at ABIHandler.commit (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/abi_handler/abi_handler.js:316:36)
    at async Generator._generateBlock (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/generator.js:396:31)
    at async Generator._generateLoop (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/generator/generator.js:299:32)
    at async Timeout._onTimeout (/home/lisk/lisk-core/node_modules/@liskhq/lisk-utils/dist-node/job_handlers/scheduler.js:26:17)] Failed to generate a block
2023-06-13T09:15:31.385Z ERROR betanet engine 593418 [peerId=80.211.194.139:7667 err=Apply penalty and abort synchronization mechanism with reason: Peer didn't return a common block trace=ApplyPenaltyAndAbortError: Apply penalty and abort synchronization mechanism with reason: Peer didn't return a common block
    at new SynchronizerError (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/errors.js:5:1)
    at new ApplyPenaltyAndAbortError (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/errors.js:35:9)
    at FastChainSwitchingMechanism._queryBlocks (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/fast_chain_switching_mechanism.js:61:19)
    at FastChainSwitchingMechanism.run (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/fast_chain_switching_mechanism.js:15:35)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async /home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/synchronizer.js:47:13
    at async Mutex.runExclusive (/home/lisk/lisk-core/node_modules/@liskhq/lisk-utils/dist-node/job_handlers/mutex.js:23:20)
    at async Synchronizer.run (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/synchronizer/synchronizer.js:33:9)
    at async Consensus._sync (/home/lisk/lisk-core/node_modules/lisk-framework/dist-node/engine/consensus/consensus.js:584:13)] Fail to handle received block
/home/lisk/lisk-core/bin/lisk-core: line 54: 593418 Killed                  "$NODE" "$DIR/run" "$@"

@punkrock
Copy link
Author

Not sure if it is an important info but I don't have to restart lisk core to fix it. I just have to re-enable the generator. Do you still need more logs or is it clear to you what the issue is, @ManuGowda?

@shuse2 shuse2 self-assigned this Jun 14, 2023
@shuse2 shuse2 added this to the Sprint 97 milestone Jun 14, 2023
@shuse2
Copy link
Collaborator

shuse2 commented Jun 14, 2023

@punkrock re-enabling generator will fix the issue seems bit strange , but it looks like somehow your node is disconnected from the network.
It seems like somehow the node is getting banned or not being able to connect for syncing 🤔

I'm investigating further

@punkrock
Copy link
Author

@shuse2 Just to make sure that there is no misunderstanding: With "fix the issue" I mean that I can re-enable the generator without the need to restart Lisk Core. Of course it doesn't fix anything because it happens some hours later again. Something is just disabling the generator for what ever reason.

@punkrock
Copy link
Author

Ok, new infos @sameersubudhi @shuse2 @ManuGowda: The reason why I am starting to miss blocks is because Lisk Core restarts itself for unknown reason and after a restart the generator is obviously disabled. So, we need to figure out, why Lisk Core restarts itself. Here are the logs from Core just before it restarted:

lisk-core-out.log

...
2023-06-14T15:10:36.029Z TRACE betanet engine 650882 [validatorAddress=lskeqhjv84465mtjqtmpthkzuucovd7nmrwa7y5r4 height=71791] Received single commit is invalid
2023-06-14T15:10:36.196Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71791] Received single commit is invalid
2023-06-14T15:10:36.369Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71894] Received single commit is invalid
2023-06-14T15:10:36.526Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71894] Received single commit is invalid
2023-06-14T15:10:36.991Z TRACE betanet engine 650882 [peerId=95.179.146.71:7667 ipAddress=95.179.146.71 port=7667] EVENT_UPDATED_PEER_INFO: Update peer info
2023-06-14T15:10:37.451Z TRACE betanet engine 650882 [peerId=95.179.146.71:7667 event=postNodeInfo] EVENT_MESSAGE_RECEIVED: Received inbound message
2023-06-14T15:10:37.562Z DEBUG betanet engine 650882 [peerId=95.179.146.71:7667] Received new node info
2023-06-14T15:10:38.156Z TRACE betanet engine 650882 [peerId=95.179.146.71:7667 event=postBlock] EVENT_MESSAGE_RECEIVED: Received inbound message
2023-06-14T15:10:39.620Z TRACE betanet engine 650882 [peerId=95.179.146.71:7667 ipAddress=95.179.146.71 port=7667] EVENT_UPDATED_PEER_INFO: Update peer info
2023-06-14T15:10:39.923Z TRACE betanet engine 650882 [peerId=95.179.146.71:7667 event=postNodeInfo] EVENT_MESSAGE_RECEIVED: Received inbound message
2023-06-14T15:10:40.920Z DEBUG betanet engine 650882 [peerId=95.179.146.71:7667] Received new node info
2023-06-14T15:10:41.342Z TRACE betanet engine 650882 [peerId=95.179.146.71:7667 event=postSingleCommitMessages] EVENT_MESSAGE_RECEIVED: Received inbound message
2023-06-14T15:10:41.998Z INFO betanet engine 650882 [status=success event=network_newBlock] IPCServer published event
2023-06-14T15:10:42.198Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69113] Received single commit is invalid
2023-06-14T15:10:42.356Z INFO betanet engine 650882 [status=success event=network_newBlock] WSServer published event
2023-06-14T15:10:42.515Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69216] Received single commit is invalid
2023-06-14T15:10:42.761Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69319] Received single commit is invalid
2023-06-14T15:10:42.969Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69422] Received single commit is invalid
2023-06-14T15:10:43.170Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=69525] Received single commit is invalid
2023-06-14T15:10:43.339Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69525] Received single commit is invalid
2023-06-14T15:10:43.480Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69628] Received single commit is invalid
2023-06-14T15:10:43.703Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=69628] Received single commit is invalid
2023-06-14T15:10:43.898Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69731] Received single commit is invalid
2023-06-14T15:10:44.101Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69731] Received single commit is invalid
2023-06-14T15:10:44.299Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69834] Received single commit is invalid
2023-06-14T15:10:44.544Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69834] Received single commit is invalid
2023-06-14T15:10:44.731Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69937] Received single commit is invalid
2023-06-14T15:10:44.916Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69937] Received single commit is invalid
2023-06-14T15:10:45.092Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70040] Received single commit is invalid
2023-06-14T15:10:45.311Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70040] Received single commit is invalid
2023-06-14T15:10:45.562Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70143] Received single commit is invalid
2023-06-14T15:10:45.845Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70143] Received single commit is invalid
2023-06-14T15:10:46.074Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70246] Received single commit is invalid
2023-06-14T15:10:46.362Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70246] Received single commit is invalid
2023-06-14T15:10:46.612Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70246] Received single commit is invalid
2023-06-14T15:10:46.992Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70349] Received single commit is invalid
2023-06-14T15:10:47.240Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70452] Received single commit is invalid
2023-06-14T15:10:47.541Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70452] Received single commit is invalid
2023-06-14T15:10:47.808Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70452] Received single commit is invalid
2023-06-14T15:10:48.023Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70555] Received single commit is invalid
2023-06-14T15:10:48.222Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70658] Received single commit is invalid
2023-06-14T15:10:48.425Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70761] Received single commit is invalid
2023-06-14T15:10:48.651Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70864] Received single commit is invalid
2023-06-14T15:10:48.796Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70967] Received single commit is invalid
2023-06-14T15:10:48.945Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71070] Received single commit is invalid
2023-06-14T15:10:49.078Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71173] Received single commit is invalid
2023-06-14T15:10:49.252Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71276] Received single commit is invalid
2023-06-14T15:10:49.450Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71379] Received single commit is invalid
2023-06-14T15:10:49.612Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71482] Received single commit is invalid
2023-06-14T15:10:49.792Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71482] Received single commit is invalid
2023-06-14T15:10:49.987Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71585] Received single commit is invalid
2023-06-14T15:10:50.185Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71585] Received single commit is invalid
2023-06-14T15:10:50.373Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71688] Received single commit is invalid
2023-06-14T15:10:50.540Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71688] Received single commit is invalid
2023-06-14T15:10:50.681Z TRACE betanet engine 650882 [validatorAddress=lskeqhjv84465mtjqtmpthkzuucovd7nmrwa7y5r4 height=71791] Received single commit is invalid
2023-06-14T15:10:50.985Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71791] Received single commit is invalid
2023-06-14T15:10:51.198Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71894] Received single commit is invalid
2023-06-14T15:10:51.373Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71894] Received single commit is invalid
2023-06-14T15:10:51.806Z TRACE betanet engine 650882 [peerId=94.130.98.238:7667 event=postSingleCommitMessages] EVENT_MESSAGE_RECEIVED: Received inbound message
2023-06-14T15:10:52.276Z TRACE betanet engine 650882 [ipAddress=94.130.98.238 port=7667 peerId=94.130.98.238:7667] EVENT_UPDATED_PEER_INFO: Update peer info
2023-06-14T15:10:52.518Z TRACE betanet engine 650882 [peerId=94.130.98.238:7667 event=postNodeInfo] EVENT_MESSAGE_RECEIVED: Received inbound message
2023-06-14T15:10:52.632Z DEBUG betanet engine 650882 [peerId=94.130.98.238:7667] Received new node info
2023-06-14T15:10:53.098Z TRACE betanet engine 650882 [peerId=94.130.98.238:7667 event=postBlock] EVENT_MESSAGE_RECEIVED: Received inbound message
2023-06-14T15:10:54.337Z TRACE betanet engine 650882 [ipAddress=94.130.98.238 port=7667 peerId=94.130.98.238:7667] EVENT_UPDATED_PEER_INFO: Update peer info
2023-06-14T15:10:54.597Z TRACE betanet engine 650882 [peerId=94.130.98.238:7667 event=postNodeInfo] EVENT_MESSAGE_RECEIVED: Received inbound message
2023-06-14T15:10:54.692Z DEBUG betanet engine 650882 [peerId=94.130.98.238:7667] Received new node info
2023-06-14T15:10:55.030Z TRACE betanet engine 650882 [peerId=94.130.98.238:7667 event=postSingleCommitMessages] EVENT_MESSAGE_RECEIVED: Received inbound message
2023-06-14T15:10:57.328Z TRACE betanet engine 650882 [peerId=94.130.98.238:7667 event=postSingleCommitMessages] EVENT_MESSAGE_RECEIVED: Received inbound message
2023-06-14T15:10:59.863Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69113] Received single commit is invalid
2023-06-14T15:11:00.077Z INFO betanet engine 650882 [status=success event=network_newBlock] IPCServer published event
2023-06-14T15:11:00.262Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69113] Received single commit is invalid
2023-06-14T15:11:00.456Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69113] Received single commit is invalid
2023-06-14T15:11:00.698Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69216] Received single commit is invalid
2023-06-14T15:11:00.845Z INFO betanet engine 650882 [status=success event=network_newBlock] WSServer published event
2023-06-14T15:11:01.012Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69216] Received single commit is invalid
2023-06-14T15:11:01.184Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69216] Received single commit is invalid
2023-06-14T15:11:01.381Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69319] Received single commit is invalid
2023-06-14T15:11:01.550Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69319] Received single commit is invalid
2023-06-14T15:11:01.804Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69319] Received single commit is invalid
2023-06-14T15:11:01.978Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69422] Received single commit is invalid
2023-06-14T15:11:02.219Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69422] Received single commit is invalid
2023-06-14T15:11:02.399Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69422] Received single commit is invalid
2023-06-14T15:11:02.621Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=69525] Received single commit is invalid
2023-06-14T15:11:02.841Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=69525] Received single commit is invalid
2023-06-14T15:11:03.052Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=69525] Received single commit is invalid
2023-06-14T15:11:03.279Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69525] Received single commit is invalid
2023-06-14T15:11:03.474Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69525] Received single commit is invalid
2023-06-14T15:11:03.683Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69525] Received single commit is invalid
2023-06-14T15:11:03.890Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69628] Received single commit is invalid
2023-06-14T15:11:04.184Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69628] Received single commit is invalid
2023-06-14T15:11:04.390Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69628] Received single commit is invalid
2023-06-14T15:11:04.549Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=69628] Received single commit is invalid
2023-06-14T15:11:04.714Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=69628] Received single commit is invalid
2023-06-14T15:11:04.879Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=69628] Received single commit is invalid
2023-06-14T15:11:05.082Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69731] Received single commit is invalid
2023-06-14T15:11:05.271Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69731] Received single commit is invalid
2023-06-14T15:11:05.450Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69731] Received single commit is invalid
2023-06-14T15:11:05.653Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69731] Received single commit is invalid
2023-06-14T15:11:05.859Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69731] Received single commit is invalid
2023-06-14T15:11:06.037Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69731] Received single commit is invalid
2023-06-14T15:11:06.222Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69834] Received single commit is invalid
2023-06-14T15:11:06.396Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69834] Received single commit is invalid
2023-06-14T15:11:06.575Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69834] Received single commit is invalid
2023-06-14T15:11:06.803Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69834] Received single commit is invalid
2023-06-14T15:11:06.966Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69834] Received single commit is invalid
2023-06-14T15:11:07.110Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69834] Received single commit is invalid
2023-06-14T15:11:07.270Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69937] Received single commit is invalid
2023-06-14T15:11:07.425Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69937] Received single commit is invalid
2023-06-14T15:11:07.623Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=69937] Received single commit is invalid
2023-06-14T15:11:07.790Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69937] Received single commit is invalid
2023-06-14T15:11:08.110Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69937] Received single commit is invalid
2023-06-14T15:11:08.281Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=69937] Received single commit is invalid
2023-06-14T15:11:08.449Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70040] Received single commit is invalid
2023-06-14T15:11:08.653Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70040] Received single commit is invalid
2023-06-14T15:11:08.879Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70040] Received single commit is invalid
2023-06-14T15:11:09.089Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70040] Received single commit is invalid
2023-06-14T15:11:09.263Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70040] Received single commit is invalid
2023-06-14T15:11:09.452Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70040] Received single commit is invalid
2023-06-14T15:11:09.632Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70143] Received single commit is invalid
2023-06-14T15:11:09.872Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70143] Received single commit is invalid
2023-06-14T15:11:10.111Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70143] Received single commit is invalid
2023-06-14T15:11:10.328Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70143] Received single commit is invalid
2023-06-14T15:11:10.525Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70143] Received single commit is invalid
2023-06-14T15:11:10.759Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70143] Received single commit is invalid
2023-06-14T15:11:10.985Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70246] Received single commit is invalid
2023-06-14T15:11:11.293Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70246] Received single commit is invalid
2023-06-14T15:11:12.548Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70246] Received single commit is invalid
2023-06-14T15:11:12.724Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70246] Received single commit is invalid
2023-06-14T15:11:12.929Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70246] Received single commit is invalid
2023-06-14T15:11:13.147Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70246] Received single commit is invalid
2023-06-14T15:11:13.393Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70246] Received single commit is invalid
2023-06-14T15:11:13.577Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70246] Received single commit is invalid
2023-06-14T15:11:13.770Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70246] Received single commit is invalid
2023-06-14T15:11:13.954Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70349] Received single commit is invalid
2023-06-14T15:11:14.137Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70349] Received single commit is invalid
2023-06-14T15:11:14.381Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70349] Received single commit is invalid
2023-06-14T15:11:14.600Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70452] Received single commit is invalid
2023-06-14T15:11:14.800Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70452] Received single commit is invalid
2023-06-14T15:11:15.020Z TRACE betanet engine 650882 [validatorAddress=lskhf3gtj3wecxaxy7csqb5k8kszugghpscvat5b4 height=70452] Received single commit is invalid
2023-06-14T15:11:15.244Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70452] Received single commit is invalid
2023-06-14T15:11:15.468Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70452] Received single commit is invalid
2023-06-14T15:11:15.692Z TRACE betanet engine 650882 [validatorAddress=lska7owayy5cgmko5gxxou9kgf6gjmharb94k34rv height=70452] Received single commit is invalid
2023-06-14T15:11:15.880Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70452] Received single commit is invalid
2023-06-14T15:11:16.044Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70452] Received single commit is invalid
2023-06-14T15:11:16.209Z TRACE betanet engine 650882 [validatorAddress=lskmfoq8b2xjk9ttnfhhhjq33ukas4ndb7eckc3c3 height=70452] Received single commit is invalid
2023-06-14T15:11:16.369Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70555] Received single commit is invalid
2023-06-14T15:11:16.515Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70555] Received single commit is invalid
2023-06-14T15:11:16.638Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70555] Received single commit is invalid
2023-06-14T15:11:16.769Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70658] Received single commit is invalid
2023-06-14T15:11:16.902Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70658] Received single commit is invalid
2023-06-14T15:11:17.130Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70658] Received single commit is invalid
2023-06-14T15:11:17.379Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70761] Received single commit is invalid
2023-06-14T15:11:17.607Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70761] Received single commit is invalid
2023-06-14T15:11:17.806Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70761] Received single commit is invalid
2023-06-14T15:11:18.053Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70864] Received single commit is invalid
2023-06-14T15:11:18.374Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70864] Received single commit is invalid
2023-06-14T15:11:18.486Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70864] Received single commit is invalid
2023-06-14T15:11:18.727Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70967] Received single commit is invalid
2023-06-14T15:11:18.989Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70967] Received single commit is invalid
2023-06-14T15:11:19.127Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=70967] Received single commit is invalid
2023-06-14T15:11:19.273Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71070] Received single commit is invalid
2023-06-14T15:11:19.408Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71070] Received single commit is invalid
2023-06-14T15:11:19.564Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71070] Received single commit is invalid
2023-06-14T15:11:19.676Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71173] Received single commit is invalid
2023-06-14T15:11:19.783Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71173] Received single commit is invalid
2023-06-14T15:11:19.922Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71173] Received single commit is invalid
2023-06-14T15:11:20.051Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71276] Received single commit is invalid
2023-06-14T15:11:20.180Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71276] Received single commit is invalid
2023-06-14T15:11:20.308Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71276] Received single commit is invalid
2023-06-14T15:11:20.437Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71379] Received single commit is invalid
2023-06-14T15:11:20.563Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71379] Received single commit is invalid
2023-06-14T15:11:20.690Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71379] Received single commit is invalid
2023-06-14T15:11:20.812Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71482] Received single commit is invalid
2023-06-14T15:11:20.987Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71482] Received single commit is invalid
2023-06-14T15:11:21.205Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71482] Received single commit is invalid
2023-06-14T15:11:21.420Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71482] Received single commit is invalid
2023-06-14T15:11:21.628Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71482] Received single commit is invalid
2023-06-14T15:11:21.831Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71482] Received single commit is invalid
2023-06-14T15:11:22.037Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71585] Received single commit is invalid
2023-06-14T15:11:22.231Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71585] Received single commit is invalid
2023-06-14T15:11:22.478Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71585] Received single commit is invalid
2023-06-14T15:11:22.728Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71585] Received single commit is invalid
2023-06-14T15:11:22.904Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71585] Received single commit is invalid
2023-06-14T15:11:23.115Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71585] Received single commit is invalid
2023-06-14T15:11:23.303Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71688] Received single commit is invalid
2023-06-14T15:11:23.531Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71688] Received single commit is invalid
2023-06-14T15:11:23.763Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71688] Received single commit is invalid
2023-06-14T15:11:23.972Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71688] Received single commit is invalid
2023-06-14T15:11:24.151Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71688] Received single commit is invalid
2023-06-14T15:11:24.360Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71688] Received single commit is invalid
2023-06-14T15:11:24.542Z TRACE betanet engine 650882 [validatorAddress=lskeqhjv84465mtjqtmpthkzuucovd7nmrwa7y5r4 height=71791] Received single commit is invalid
2023-06-14T15:11:24.755Z TRACE betanet engine 650882 [validatorAddress=lskeqhjv84465mtjqtmpthkzuucovd7nmrwa7y5r4 height=71791] Received single commit is invalid
2023-06-14T15:11:24.969Z TRACE betanet engine 650882 [validatorAddress=lskeqhjv84465mtjqtmpthkzuucovd7nmrwa7y5r4 height=71791] Received single commit is invalid
2023-06-14T15:11:25.193Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71791] Received single commit is invalid
2023-06-14T15:11:25.392Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71791] Received single commit is invalid
2023-06-14T15:11:25.606Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71791] Received single commit is invalid
2023-06-14T15:11:25.806Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71894] Received single commit is invalid
2023-06-14T15:11:26.040Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71894] Received single commit is invalid
2023-06-14T15:11:26.223Z TRACE betanet engine 650882 [validatorAddress=lskmz22uzabdhhad2ddb2f6hqky9v9b4n926hggfg height=71894] Received single commit is invalid
2023-06-14T15:11:26.406Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71894] Received single commit is invalid
2023-06-14T15:11:26.591Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71894] Received single commit is invalid
2023-06-14T15:11:26.741Z TRACE betanet engine 650882 [validatorAddress=lskbm4p4sava9x75osfcw67jwv387m7zgwvz35mkw height=71894] Received single commit is invalid
2023-06-14T15:11:28.451Z DEBUG betanet engine 650882 [currentSlot=168675503] Waiting for validator slot


Starting Lisk lisk-core at /home/lisk/.lisk/lisk-core.


2023-06-14T15:11:53.371Z INFO betanet application 655949 Starting the app at /home/lisk/.lisk/lisk-core
2023-06-14T15:11:53.371Z INFO betanet application 655949 If you experience any type of error, please open an issue on Lisk GitHub: https://github.com/LiskHQ/lisk-sdk/issues
2023-06-14T15:11:53.371Z INFO betanet application 655949 Contribution guidelines can be found at Lisk-sdk: https://github.com/LiskHQ/lisk-sdk/blob/development/docs/CONTRIBUTING.md
2023-06-14T15:11:53.371Z INFO betanet application 655949 Booting the application with Lisk Framework
2023-06-14T15:11:53.410Z INFO betanet application 650882 Previous Lisk PID
2023-06-14T15:11:53.410Z INFO betanet application 655949 Current Lisk PID
...

@punkrock
Copy link
Author

punkrock commented Jun 14, 2023 •
edited
Loading

Another update. From my syslogs. looks like 2 GB RAM without SWAP are not enough. I will add a SWAP of 4 GB and see if it works:

Jun 14 15:11:51 betanet kernel: [1903749.357609] node /home/lisk invoked oom-killer: gfp_mask=0x1100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
Jun 14 15:11:51 betanet kernel: [1903749.357615] CPU: 1 PID: 100948 Comm: node /home/lisk Not tainted 5.15.0-70-generic #77-Ubuntu
Jun 14 15:11:51 betanet kernel: [1903749.357618] Hardware name: Hetzner vServer/Standard PC (Q35 + ICH9, 2009), BIOS 20171111 11/11/2017
Jun 14 15:11:51 betanet kernel: [1903749.357619] Call Trace:
Jun 14 15:11:51 betanet kernel: [1903749.357621]  <TASK>
Jun 14 15:11:51 betanet kernel: [1903749.357623]  show_stack+0x52/0x5c
Jun 14 15:11:51 betanet kernel: [1903749.357628]  dump_stack_lvl+0x4a/0x63
Jun 14 15:11:51 betanet kernel: [1903749.357631]  dump_stack+0x10/0x16
Jun 14 15:11:51 betanet kernel: [1903749.357632]  dump_header+0x53/0x228
Jun 14 15:11:51 betanet kernel: [1903749.357635]  oom_kill_process.cold+0xb/0x10
Jun 14 15:11:51 betanet kernel: [1903749.357637]  out_of_memory+0x106/0x2e0
Jun 14 15:11:51 betanet kernel: [1903749.357641]  __alloc_pages_slowpath.constprop.0+0x9b7/0xa80
Jun 14 15:11:51 betanet kernel: [1903749.357644]  __alloc_pages+0x311/0x330
Jun 14 15:11:51 betanet kernel: [1903749.357646]  alloc_pages+0x9e/0x1e0
Jun 14 15:11:51 betanet kernel: [1903749.357648]  __page_cache_alloc+0x7e/0x90
Jun 14 15:11:51 betanet kernel: [1903749.357650]  pagecache_get_page+0x152/0x590
Jun 14 15:11:51 betanet kernel: [1903749.357652]  ? page_cache_ra_unbounded+0x166/0x210
Jun 14 15:11:51 betanet kernel: [1903749.357654]  filemap_fault+0x488/0xab0
Jun 14 15:11:51 betanet kernel: [1903749.357655]  ? filemap_map_pages+0x309/0x400
Jun 14 15:11:51 betanet kernel: [1903749.357657]  __do_fault+0x3c/0x120
Jun 14 15:11:51 betanet kernel: [1903749.357659]  do_read_fault+0xeb/0x160
Jun 14 15:11:51 betanet kernel: [1903749.357661]  do_fault+0xa0/0x2e0
Jun 14 15:11:51 betanet kernel: [1903749.357663]  handle_pte_fault+0x1cd/0x240
Jun 14 15:11:51 betanet kernel: [1903749.357665]  __handle_mm_fault+0x405/0x6f0
Jun 14 15:11:51 betanet kernel: [1903749.357667]  handle_mm_fault+0xd8/0x2c0
Jun 14 15:11:51 betanet kernel: [1903749.357669]  do_user_addr_fault+0x1c9/0x670
Jun 14 15:11:51 betanet kernel: [1903749.357672]  exc_page_fault+0x77/0x170
Jun 14 15:11:51 betanet kernel: [1903749.357674]  asm_exc_page_fault+0x27/0x30
Jun 14 15:11:51 betanet kernel: [1903749.357677] RIP: 0033:0x1d9b9f0
Jun 14 15:11:51 betanet kernel: [1903749.357684] Code: Unable to access opcode bytes at RIP 0x1d9b9c6.
Jun 14 15:11:51 betanet kernel: [1903749.357685] RSP: 002b:00007ffd6f51c5d0 EFLAGS: 00010293
Jun 14 15:11:51 betanet kernel: [1903749.357688] RAX: 00000000000000c1 RBX: 00007ffd6f51c700 RCX: 00007ffd6f51c8e0
Jun 14 15:11:51 betanet kernel: [1903749.357689] RDX: 00007ffd6f51c958 RSI: 000000000000005c RDI: 00007ffd6f51c7e0
Jun 14 15:11:51 betanet kernel: [1903749.357691] RBP: 00007ffd6f51c5e0 R08: 0000000000000061 R09: 0000000005d43278
Jun 14 15:11:51 betanet kernel: [1903749.357692] R10: 0000000000000000 R11: 0000000000000000 R12: 000000000000005c
Jun 14 15:11:51 betanet kernel: [1903749.357693] R13: 00007ffd6f51c700 R14: 0000000000000003 R15: 000000007fffffff
Jun 14 15:11:51 betanet kernel: [1903749.357695]  </TASK>
Jun 14 15:11:51 betanet kernel: [1903749.357696] Mem-Info:
Jun 14 15:11:51 betanet kernel: [1903749.357697] active_anon:165 inactive_anon:444558 isolated_anon:0
Jun 14 15:11:51 betanet kernel: [1903749.357697]  active_file:23 inactive_file:4 isolated_file:0
Jun 14 15:11:51 betanet kernel: [1903749.357697]  unevictable:7681 dirty:0 writeback:0
Jun 14 15:11:51 betanet kernel: [1903749.357697]  slab_reclaimable:5924 slab_unreclaimable:8340
Jun 14 15:11:51 betanet kernel: [1903749.357697]  mapped:2287 shmem:980 pagetables:2443 bounce:0
Jun 14 15:11:51 betanet kernel: [1903749.357697]  kernel_misc_reclaimable:0
Jun 14 15:11:51 betanet kernel: [1903749.357697]  free:16693 free_pcp:624 free_cma:0
Jun 14 15:11:51 betanet kernel: [1903749.357701] Node 0 active_anon:660kB inactive_anon:1778232kB active_file:92kB inactive_file:16kB unevictable:30724kB isolated(anon):0kB isolated(file):0kB mapped:9148kB dirty:0kB writeback:0kB shmem:3920kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:3056kB pagetables:9772kB all_unreclaimable? no
Jun 14 15:11:51 betanet kernel: [1903749.357705] Node 0 DMA free:7812kB min:356kB low:444kB high:532kB reserved_highatomic:0KB active_anon:0kB inactive_anon:7192kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:12kB local_pcp:12kB free_cma:0kB
Jun 14 15:11:51 betanet kernel: [1903749.357709] lowmem_reserve[]: 0 1866 1866 1866 1866
Jun 14 15:11:51 betanet kernel: [1903749.357711] Node 0 DMA32 free:58960kB min:44696kB low:55868kB high:67040kB reserved_highatomic:14336KB active_anon:660kB inactive_anon:1771040kB active_file:324kB inactive_file:16kB unevictable:30724kB writepending:0kB present:2031472kB managed:1958616kB mlocked:27652kB bounce:0kB free_pcp:2484kB local_pcp:2220kB free_cma:0kB
Jun 14 15:11:51 betanet kernel: [1903749.357715] lowmem_reserve[]: 0 0 0 0 0
Jun 14 15:11:51 betanet kernel: [1903749.357717] Node 0 DMA: 5*4kB (M) 12*8kB (UME) 9*16kB (UME) 6*32kB (UME) 3*64kB (UE) 6*128kB (UME) 3*256kB (ME) 5*512kB (UME) 1*1024kB (U) 1*2048kB (M) 0*4096kB = 7812kB
Jun 14 15:11:51 betanet kernel: [1903749.357727] Node 0 DMA32: 380*4kB (UME) 864*8kB (ME) 518*16kB (UME) 318*32kB (UME) 169*64kB (UME) 60*128kB (UME) 23*256kB (UME) 5*512kB (ME) 5*1024kB (UM) 0*2048kB 0*4096kB = 58960kB
Jun 14 15:11:51 betanet kernel: [1903749.357739] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Jun 14 15:11:51 betanet kernel: [1903749.357741] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Jun 14 15:11:51 betanet kernel: [1903749.357742] 3273 total pagecache pages
Jun 14 15:11:51 betanet kernel: [1903749.357742] 0 pages in swap cache
Jun 14 15:11:51 betanet kernel: [1903749.357743] Swap cache stats: add 0, delete 0, find 0/0
Jun 14 15:11:51 betanet kernel: [1903749.357744] Free swap  = 0kB
Jun 14 15:11:51 betanet kernel: [1903749.357745] Total swap = 0kB
Jun 14 15:11:51 betanet kernel: [1903749.357745] 511866 pages RAM
Jun 14 15:11:51 betanet kernel: [1903749.357746] 0 pages HighMem/MovableOnly
Jun 14 15:11:51 betanet kernel: [1903749.357746] 18372 pages reserved
Jun 14 15:11:51 betanet kernel: [1903749.357747] 0 pages hwpoisoned
Jun 14 15:11:51 betanet kernel: [1903749.357747] Tasks state (memory values in pages):
Jun 14 15:11:51 betanet kernel: [1903749.357748] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
Jun 14 15:11:51 betanet kernel: [1903749.357752] [    435]     0   435    72337     6775   110592        0         -1000 multipathd
Jun 14 15:11:51 betanet kernel: [1903749.357755] [    738]   103   738     2221      804    57344        0          -900 dbus-daemon
Jun 14 15:11:51 betanet kernel: [1903749.357757] [    748]     0   748     8158     2662   110592        0             0 networkd-dispat
Jun 14 15:11:51 betanet kernel: [1903749.357759] [    752]     0   752     1723      458    53248        0             0 cron
Jun 14 15:11:51 betanet kernel: [1903749.357760] [    759]     0   759     3847      550    73728        0             0 systemd-logind
Jun 14 15:11:51 betanet kernel: [1903749.357762] [    765]     0   765      965      522    45056        0             0 atd
Jun 14 15:11:51 betanet kernel: [1903749.357763] [    793]     0   793     1449      166    49152        0             0 agetty
Jun 14 15:11:51 betanet kernel: [1903749.357765] [    795]     0   795    27436     2875   114688        0             0 unattended-upgr
Jun 14 15:11:51 betanet kernel: [1903749.357766] [    797]     0   797     1543      210    53248        0             0 agetty
Jun 14 15:11:51 betanet kernel: [1903749.357768] [   7193]   107  7193    55600      826    81920        0             0 rsyslogd
Jun 14 15:11:51 betanet kernel: [1903749.357770] [  11659]     0 11659    20699      691    61440        0             0 irqbalance
Jun 14 15:11:51 betanet kernel: [1903749.357771] [  11661]     0 11661    73909      891   159744        0             0 packagekitd
Jun 14 15:11:51 betanet kernel: [1903749.357773] [  11665]     0 11665    74406      746   577536        0          -250 systemd-journal
Jun 14 15:11:51 betanet kernel: [1903749.357775] [  11667]     0 11667    58622      451    86016        0             0 polkitd
Jun 14 15:11:51 betanet kernel: [1903749.357776] [  11678]   104 11678    22339      640    69632        0             0 systemd-timesyn
Jun 14 15:11:51 betanet kernel: [1903749.357777] [  11679]     0 11679     5562      477    61440        0         -1000 systemd-udevd
Jun 14 15:11:51 betanet kernel: [1903749.357779] [  11680]     0 11680    98116     1061   131072        0             0 udisksd
Jun 14 15:11:51 betanet kernel: [1903749.357780] [  11687]   101 11687     4030      529    69632        0             0 systemd-network
Jun 14 15:11:51 betanet kernel: [1903749.357782] [  11691]   102 11691     6317     1398    81920        0             0 systemd-resolve
Jun 14 15:11:51 betanet kernel: [1903749.357783] [  21044]     0 21044     3857     1081    61440        0         -1000 sshd
Jun 14 15:11:51 betanet kernel: [1903749.357785] [ 100286]  1000 100286     4258      972    73728        0             0 systemd
Jun 14 15:11:51 betanet kernel: [1903749.357786] [ 100287]  1000 100287    42553     1189   102400        0             0 (sd-pam)
Jun 14 15:11:51 betanet kernel: [1903749.357788] [ 100898]  1000 100898   227200    10107  1155072        0             0 PM2 v5.3.0: God
Jun 14 15:11:51 betanet kernel: [1903749.357789] [ 100948]  1000 100948   234828    14700  1556480        0             0 node /home/lisk
Jun 14 15:11:51 betanet kernel: [1903749.357791] [ 650875]  1000 650875     1842      512    53248        0             0 bash
Jun 14 15:11:51 betanet kernel: [1903749.357792] [ 650882]  1000 650882   616433   405762  5931008        0             0 node
Jun 14 15:11:51 betanet kernel: [1903749.357794] [ 652005]     0 652005     4228     1236    77824        0             0 sshd
Jun 14 15:11:51 betanet kernel: [1903749.357795] [ 652084]  1000 652084     4301      877    81920        0             0 sshd
Jun 14 15:11:51 betanet kernel: [1903749.357796] [ 652085]  1000 652085     3106     1812    61440        0             0 bash
Jun 14 15:11:51 betanet kernel: [1903749.357798] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-1000.slice/session-198.scope,task=node,pid=650882,uid=1000Jun 14 15:11:51 betanet kernel: [1903749.357854] Out of memory: Killed process 650882 (node) total-vm:2465732kB, anon-rss:1623048kB, file-rss:0kB, shmem-rss:0kB, UID:1000 pgtables:5792kB oom_score_adj:0
Jun 14 15:11:51 betanet systemd[1]: session-198.scope: A process of this unit has been killed by the OOM killer.

@shuse2
Copy link
Collaborator

shuse2 commented Jun 15, 2023

@punkrock thanks for the investigation. Maybe 2GB is too small to run the Core 4.0 node.
If it goes OOM during the block execution, the node will die and it will loose the block generator information.

Let us know how it goes after changing to the 4GB memory 🙏

@Madhulearn Madhulearn mentioned this issue Jun 20, 2023
Closed
@Madhulearn Madhulearn modified the milestones: Sprint 97, Sprint 98 Jun 20, 2023
@Madhulearn Madhulearn modified the milestones: Sprint 98, Sprint 99 Jul 4, 2023
@Madhulearn Madhulearn modified the milestones: Sprint 99, Sprint 100 Jul 18, 2023
@shuse2
Copy link
Collaborator

shuse2 commented Jul 26, 2023

Since there is no update, I will assume this problem is resolved

@shuse2 shuse2 closed this as not planned Won't fix, can't repro, duplicate, stale Jul 26, 2023
@github-project-automation github-project-automation bot moved this from Backlog to Done in Lisk SDK Version 6.0.0 Jul 26, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@shuse2 shuse2

Labels
framework type: bug
Projects
None yet
Milestone
Sprint 100
Development

No branches or pull requests
5 participants
@punkrock @shuse2 @ManuGowda @sameersubudhi @Madhulearn