Collator parachain node sync errors - Digest item must match that calculated

[firke]

Description

We're using Astar Collator node as a Shiden RPC endpoint. At some moment of time chain stops syncing and block ingesting stops completely. Restarting the node fixes the issue, but it usually appears within the next 24-48h. Same issue appears throughout the sync retries - Digest item must match that calculated and Err(UnknownParent).

Steps to Reproduce

1. Node is setup.
2. Node is fully synced on both relay chain and parachain.
3. At some point of time chain sync stops.
4. Restart is required to continue syncing.

Expected vs. Actual Behavior

Would expect the chain to be able to sync without manual restarts.

Environment

• Operating system: Ubuntu 20.04.3 LTS
• Project version/tag: v2.7.1.
• Rust version (run rustup show): Using pre-compiled binary from the releases page.

Logs, Errors or Screenshots

2021-11-02 09:31:37 [Parachain] panicked at 'Digest item must match that calculated.', /cargo-home/git/checkouts/substrate-82367bbcc2c32d4b/a477f54/frame/executive/src/lib.rs:467:13    
2021-11-02 09:31:37 [Parachain] Block prepare storage changes error:
RuntimeApiError(Application(Execution(Other("Wasm execution trapped: wasm trap: unreachable\nwasm backtrace:\n    0: 0x2cad - <unknown>!rust_begin_unwind\n    1: 0x220f - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b\n    2: 0x18a4 - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a\n    3: 0x9bcc5 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h963416ce164b04e1\n    4: 0x1ab22d - <unknown>!Core_execute_block\n"))))    
2021-11-02 09:31:37 [Parachain] 💔 Error importing block 0x5b1f6cf8cca28d39466f1db9295d28a02831e62873169fbda10684058814e908: Err(Other(ClientImport("Error at calling runtime api: Execution failed: Other(\"Wasm execution trapped: wasm trap: unreachable\\nwasm backtrace:\\n    0: 0x2cad - <unknown>!rust_begin_unwind\\n    1: 0x220f - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b\\n    2: 0x18a4 - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a\\n    3: 0x9bcc5 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h963416ce164b04e1\\n    4: 0x1ab22d - <unknown>!Core_execute_block\\n\")")))    
2021-11-02 09:31:37 [Parachain] panicked at 'Digest item must match that calculated.', /cargo-home/git/checkouts/substrate-82367bbcc2c32d4b/a477f54/frame/executive/src/lib.rs:467:13    
2021-11-02 09:31:37 [Parachain] Block prepare storage changes error:
RuntimeApiError(Application(Execution(Other("Wasm execution trapped: wasm trap: unreachable\nwasm backtrace:\n    0: 0x2cad - <unknown>!rust_begin_unwind\n    1: 0x220f - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b\n    2: 0x18a4 - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a\n    3: 0x9bcc5 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h963416ce164b04e1\n    4: 0x1ab22d - <unknown>!Core_execute_block\n"))))    
2021-11-02 09:31:37 [Parachain] 💔 Error importing block 0x5b1f6cf8cca28d39466f1db9295d28a02831e62873169fbda10684058814e908: Err(Other(ClientImport("Error at calling runtime api: Execution failed: Other(\"Wasm execution trapped: wasm trap: unreachable\\nwasm backtrace:\\n    0: 0x2cad - <unknown>!rust_begin_unwind\\n    1: 0x220f - <unknown>!core::panicking::panic_fmt::h3ab5417155b7ba3b\\n    2: 0x18a4 - <unknown>!core::panicking::panic::h5bfdfaa3db9a4b4a\\n    3: 0x9bcc5 - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPallets,COnRuntimeUpgrade>::execute_block::h963416ce164b04e1\\n    4: 0x1ab22d - <unknown>!Core_execute_block\\n\")")))    
2021-11-02 09:31:37 [Parachain] 💔 Error importing block 0x6c8d6ff602303181cab5ab6e9b9a9f0751570a912917142810c30ec9eeaf4929: Err(UnknownParent)

Additional Information

Listed as 'Astar Shiden RPC' under telemetry: https://telemetry.polkadot.io/#list/0xf1cf9022c7ebb34b162d5b5e34e705a5a740b2d0ecc1009fb89023e62a488108

[h4x3rotab]

Maybe related? This only happens after the upgrade to substrate v0.9.12

paritytech/substrate#10158

[coder-chenda]

how to solve the problem?

[andabak]

Since several new versions of both Astar and Substrate came out in the meantime, this issue will be closed.