Troubleshooting the kernel

Troubleshooting from the kernel log

Usually you can watch the kernel behaviour through the kernel log. However, the default kernel log level only show the normal, warning and error kernel operating information. If you need more information. It requires to adjust the log level in the config.xml.

The default kernel log you can find in in the <the project root>/network/<network name> You can adjust the level from ERROR, WARN,INFO,DEBUG,TRACE to retrive more kernel information

    <log>
        <!--Enable/Disable logback service; if disabled, output will not be logged -->
        <log-file>true</log-file>
        <!--Sets the physical location on disk where log files will be stored.-->
        <log-path>log</log-path>
        <ROOT>WARN</ROOT>        // general log relate to the system
        <GEN>INFO</GEN>          // the kernel general operating log
        <VM>ERROR</VM>           // the log relate to the blockchain VM
        <SYNC>INFO</SYNC>        // the log relate to the network sync
        <CONS>INFO</CONS>        // the log relate to the blockchain protocol
        <DB>WARN</DB>            // the log relate to the database
        <API>INFO</API>          // the log relate to the Rpc calls, Java API and the API server
        <P2P>INFO</P2P>          // the log relate to the p2p connect information
        <TX>WARN</TX>            // the log relate to the transaction details
        <TXPOOL>WARN</TXPOOL>    // the log relate to the transaction pending pool details
    </log>

When the kernel stuck (not importing new blocks)

Usually you can see the block import log through the terminal. If you see the kernel does not have new block import, If means:

1. The network halt, it has highly possible the protocol break, or the mining/Staking power disappear.
2. The kernel is in the side-chain and cannot recover due to the deep branching, It require to revert the database back to branch point by ./aion.sh [-n network] -r <block number> then re-sync the block with the network.
3. The kernel stuck, it usually happens due to the kernel bugs, usually it can be recovered by rebooting the kernel. If you can see the obvious error in the kernel log, usually you can copy the log error and search the code section through the IDE, then you might know where the bug is.

However, if you still don't know the issue by searching the log, you might need to dig more details. dump the JVM information before reboot the kernel by the following steps:

• In the system terminal, use top or htop to identify the kernel process ID ~ PID
• Observe the process status by top -p -H, You can see the process memory usage, CPU usage in each process thread. If you see a thread occupy the heavy CPU resources in a period of time, it usually means the issue happens in that thread.
• Usejmap -dump:live,file=<file-path> <pid> To dump the JVM core information, then you can analyse the dump file through the JVM dump file tooling.
• After gathered the dump information, backup the kernel log, and then try to reboot the node.