Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Graceful shutdown of the BSC node when running as a system service (1.3.x) #2163

Closed
HarshitJoshi opened this issue Jan 17, 2024 · 5 comments
Closed

Graceful shutdown of the BSC node when running as a system service (1.3.x) #2163

HarshitJoshi opened this issue Jan 17, 2024 · 5 comments
Assignees
@zzzckck

Comments

@HarshitJoshi
Copy link

HarshitJoshi commented Jan 17, 2024 •
edited
Loading

System information

BSC-Geth version: 1.3.6
OS & Version: Debian GNU/Linux 11 (bullseye) | x86_64 GNU/Linux
OS Resources: 32 CPU / 128 gb / 10 gbit
Storage: Western Digital NVMe 6.4 TB

This is the current service file for running the node:

[Unit]
Description=Binance Geth Daemon
After=network.target

[Service]
ExecStart=/bin/sh -c '/x/bsc-geth/bsc-geth --config /x/bsc-geth/config/config.toml'

TimeoutStopFailureMode=abort
TimeoutStopSec=600
Restart=on-failure

KillSignal=SIGHUP

[Install]
WantedBy=default.target

From what I've observed, this doesn't lead to a graceful shutdown of the node service that causes a myriad of other problems (db corruption, unclean shutdowns, etc.)

Is there a recommended service file that I can use to avoid these issues, and is there anything else that I can update above to avoid them effectively?

There's an ongoing issue with the startup of both pruned/non-pruned nodes post an "unclean shutdown", even though I'm trying to gracefully shut it down using the above service file that causes the database to be fried and the nodes require a resync. (#2110)

This is getting mission critical to avoid this problem both knowingly/unknowingly as the hard fork is right around the corner without a fix/merge to the problem - #2155

Thanks for your help! Let me know if you have any questions for me.

Edit: I've tried playing with both SIGTERM vs SIGHUP but without any noticeable changes - would like to know if there's a "recommended" less violet kill signal preferred over the other.
@HarshitJoshi HarshitJoshi changed the title Graceful shutdown of the node when running as a system service (1.3.x) Graceful shutdown of the BSC node when running as a system service (1.3.x) Jan 17, 2024
@deepcrazy
Copy link

Hi there, Please upgrade to geth v1.3.7 (https://github.com/bnb-chain/bsc/releases/tag/v1.3.7) and see if it works.

In addition to this, once upgraded, please do share the logs as well while starting the node and also when stopping the node.

@zzzckck
Copy link
Collaborator

zzzckck commented Jan 18, 2024

Basically, bsc node can be shutdown gracefully, it could take around 15second as I know. But we did notice some nodes failed to graceful shutdown due to the incorrect use of this flag: "--history.transactions 0", if you just upgrade from v1.2.x to v1.3.0, pls notice that "--txlookuplimit" has been replaced by "--history.transactions" since v1.3.x

Here is the service file and script that we use to start/stop bsc node, for your reference.
bsc.service

[Unit]
Description=bsc
After=network.target

[Service]
Type=simple
User=root
Group=root
ExecStart=/server/node/chaind.sh -start
ExecReload=/server/node/chaind.sh -restart
ExecStop=/server/node/chaind.sh -stop
PrivateTmp=true
Restart=always
LimitNOFILE=500000
RestartSec=5
TimeoutStopSec=120
StartLimitInterval=0

[Install]

/server/node/chaind.sh

#!/bin/bash
export GOGC=100

function startChaind() {
    workspace=/server/node
    ${workspace}/bsc --cache 18000 --history.transactions 0  --datadir ${workspace}/  --config ${workspace}/config.toml >> ${workspace}/bscnode.log 2>&1
}

function stopChaind() {
    pid=`ps -ef | grep /server/node/bsc | grep -v grep | awk '{print $2}'`
    if [ -n "$pid" ]; then
        kill -TERM $pid
        for((i=1;i<=40;i++));
        do
            pid=`ps -ef | grep /server/node/bsc | grep -v grep | awk '{print $2}'`
            if [ -z "$pid" ]; then
                break
            fi
            sleep 10
        done
    fi
}

CMD=$1

case $CMD in
-start)
    echo "start"
    startChaind
    ;;
-stop)
    echo "stop"
    stopChaind
    ;;
-restart)
    stopChaind
    sleep 3
    startChaind
    ;;
*)
    echo "Usage: chaind.sh -start | -stop | -restart .Or use systemctl start | stop | restart bsc.service "
    ;;
esac

@zzzckck zzzckck self-assigned this Jan 18, 2024
@HarshitJoshi
Copy link
Author

@deepcrazy - I'll try to get those logs when this happens again, unfortunately, the node logs have rotated for me to get that information, Although, the behavior between different versions wouldn't/shouldn't matter as far as initialization or breakdown of the process itself, but, I could be wrong.

@zzzckck - thanks for your input, the major differences worth highlighting that maybe(?) make the difference between graceful shutdown are as follows:

-SIGTERM (kill -TERM $pid) to stop the process over SIGHUP

  • RestartSec delay between unexpected service termination and subsequent restart attempts
  • PrivateTmp to isolate the process to an isolated directory enhancing security and minimizing the risk of unintended interference with other services or processes

Based on the above, I've tried combining the two below:

[Unit]
Description=Binance Geth Daemon
After=network.target

[Service]
ExecStart=/bin/sh -c '/x/bsc-geth/bsc-geth --config /x/bsc-geth/config/config.toml'
ExecStop=/bin/kill -15 $MAINPID

PrivateTmp=true
TimeoutStopSec=300

Restart=always
RestartSec=30
StartLimitInterval=0
StartLimitBurst=10

KillSignal=SIGTERM

[Install]
WantedBy=default.target

As to your other point, I'm not utilizing either""--history.transactions 0" or "--txloouplimit". Let me know what you think of the above or any further recommendations - otherwise, I'll just try this and force simulate a restart with the hopes of fixing my problem.

@HarshitJoshi
Copy link
Author

HarshitJoshi commented Jan 19, 2024 •
edited
Loading

I was able to reproduce the same issue, the node was stuck (#2145, #2104) based on the recommendation to remove the "nodekey" file from the downloaded snapshot or restarting the node - the db got fried, but I was able to capture the logs:

After systemctl stop of the service

Logs 
t=2024-01-18T23:44:40-0500 lvl=info msg="Got interrupt, shutting down..."
t=2024-01-18T23:44:40-0500 lvl=dbug msg="RPC server shutting down"
t=2024-01-18T23:44:40-0500 lvl=info msg="HTTP server stopped"                    endpoint=[::]:8545
t=2024-01-18T23:44:40-0500 lvl=dbug msg="RPC server shutting down"
t=2024-01-18T23:44:40-0500 lvl=info msg="HTTP server stopped"                    endpoint=[::]:8546
t=2024-01-18T23:44:40-0500 lvl=warn msg="Already shutting down, interrupt more to panic." times=9
t=2024-01-18T23:44:40-0500 lvl=dbug msg="RPC server shutting down"
t=2024-01-18T23:44:40-0500 lvl=info msg="IPC endpoint closed"                    url=/x/binance-smart-chain/bsc-geth/geth.ipc
t=2024-01-18T23:44:40-0500 lvl=dbug msg="RPC server shutting down"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="RPC connection read error"              err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=5d4319fe3a7bb67a0b7596205dd1c26ed861e9fc94dc4414819e40e0903c2bb8 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=83cf1330fd90f9fe312fb128a0a5104b747d610c4d7b1a01d93420bb8a306738 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=56c3c8b24412acd83842548954398502a4e15a58ea1f9c2fe1b741c3cbe03e04 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=5f5467456d15b4337ba4b0080b26fab5337c29bc32fa779582cd6e7bdc65a0c1 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=2a02c0d7c83cf2d5cbef13fa6f75e368ca65c13619f8d5e4261d5c9907daf32c conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=fdc66437be025c9738972a267fca1e30a545bb7c389b2d560efba89a0cfc047c conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=fdc66437 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=ca6c4f085adac072e2cfb016d8b18e08214e061915a7e2caea2d456bca83285b conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=75455dfb9bc5706029dcfc3cdb99bb008e1ee9026ba3eab18d89732340711538 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=5f247109800ea27d58a6435c15bcff69e1ad7f088647c40ce35c2869413c90b1 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=746d159c95ab8b78502ce331b8495c73734602593ee26fda3806d2242e53b352 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=742bd6eeb95216f18e31cce613b16b5965a0ba031a39118d2e5cb878a59c4576 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=83ed420d5b84db2da06ae78179cc757f0ce6e2f68f822a18fe158d1a78a2e267 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=757604906174dec7945059990ea58a7246629e42d54bdd0a45da6cee99fc3129 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=75760490 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=5f546745 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=83ed420d snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=eb89f99c42c6c9de42c8d55679a35d6ca1c4974c2defb6dcadc463b970465354 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=eb89f99c snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=7c6dee295962fa63f583b1b9166046a275a759591349a35e35073059f1ae4b67 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=7c6dee29 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=c0b37c61ac28aa589577e85858b31b93c528fb4cf9ed88ffcc1d43009bc22451 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=c0b37c61 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=ca92bde8cbfc9601a190df34ffb426f750f7a17d801708524aa3160c9cc58274 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=ca92bde8 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=56c3c8b2 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=66dcc30d16a08153020130bfa1cce7309308e68013638f3fb32d9ecfffc085e0 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=66dcc30d snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=ca0d7147d37bafede7cb71979511fc7915ce484b318141cb88922a43fb73eed0 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=ca0d7147 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=66361b67cff7c5fedc5c202da206e586be711192ba7f8bf28f56f08fb38072e7 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=66361b67 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=5d528815149e16f25c22df6feae8ac9f2689f4ef8e15528564810f76ae06a4ea conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=5d528815 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=75c5d3bdf58a440d88f1e91c23e23c4a4a9da07f9884b8a79682e8a3d83eeb05 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=75c5d3bd snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=7786caabe4cfc70db7b00f0d5dca0e30faeaa829dea27d969a828d65df89db44 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=ca7e690ecd826556ae468b99630a5c11b6b3c3432acba2f73b82b9b1de64cecc conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=ca7e690e snap=false
t=2024-01-18T23:44:40-0500 lvl=info msg="Ethereum protocol stopped"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=83ef8e00b4dbb60fceff117e4fad34f7d25bb6ab2154bb9d9a85ef02d12caee4 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=83ef8e00 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=56b66ced52e1fb2c72ffec58f00ef0c72d802ca812f1cd37e0275be63e27d2fa conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=56b66ced snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=6611eb66c811e4005445202f3641f670abc443d5b432ae9f3157e1400e3076a8 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=6611eb66 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=ca63cb63d82a53431efd577c598bca75dc4dd979f530af3aa51b3c3a41bb392b conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=ca63cb63 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=2a02c0d7 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=757bf73028a2740ff3f23c1011dc1d7ea0411739b8792c62858259cda8dbadf5 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=757bf730 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=77af1c55b38ba0b29f18edb6650c48040282dd03a31599b8856b8f427799c804 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=77af1c55 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=82677fcda0ab3dc62d9332ffd719146f18e9d710b50ca81d8b1112d9dd55e7ba conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=82677fcd snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=83156fc14ed1e5190a23dfe7ed5391a438eda29e945466c9574ffcd75c533a15 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=83156fc1 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=cb0c4d756320cff8ff70beed01f3322d4a302f96c677a8a917170975ddffdfda conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=cb0c4d75 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=5619934dcb3eca4cf9f7cae3f960826b0b7317c2081d649f474a3f64b339b9d1 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=5619934d snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=663af41b5b4bc07488d2ce431911686a469d9ed7a43ce5110874bf8d2409fde2 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=663af41b snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=cd6eebe97dd6ec2101b2208df38d2e484bf028df3e5a538e13ba9358baccb6bb conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=cd6eebe9 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=472acf2d72476621e108d49896cd112d9b2ad0b1a2223272a421d2b57fd8e1b3 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=472acf2d snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=67b66ab03da59445f452c06fbaa1980a2f6ca1965f348e78a4f5133f376f1ee5 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=67b66ab0 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=ca6c4f08 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=8254eed2cc2b046f62e386fc841785dec140b69cb67250d78c285de09b3f8022 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=8254eed2 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=75455dfb snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=578f3d0fcbe8b1df19f87423b3380a4270a96d8afcb293bfde77ad02934c7442 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=578f3d0f snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=77a409444db25cd22fdcd7925b7fc9a7dc1fd283a6e2cde09d4d9634fff0b565 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=77a40944 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=5f247109 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=565b56ff7acd2f5b73ebf3038374457a8985cc198910640547475ea9f0aeac62 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=565b56ff snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=446bd23cfaa1520f075e21ff741d8967439c3368d69d29eb660b43af3274877b conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=446bd23c snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=ca1da8cba067fec6ed866d2d907083bf0798106d8548e88c802d9afdf7fc56ac conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=ca1da8cb snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=65cad4171d9e6ba8ab2cac31b6745b1972a7a3ed8b03165088fc8c682b7cc9aa conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=65cad417 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=83cf1330 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=5d4319fe snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=561011f48ccd2f0f2e3af4bb4e505d30d54fc0a61e10209e9a6d5b3f1f329937 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=561011f4 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=5f3c26d8c9c2d2b682e88db2303fe10df67c572188c8b5438080d2286dde6893 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=5f3c26d8 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=746d159c snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=5689620ac32409653a532f6a4a936a6fc0a5d30f16a480a9cf4c3f02a0784b1a conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=5689620a snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=56e986acf6b943df92a40918127145a4e341c924edf4b6ac651b1f5912d15998 conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=56e986ac snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=fdc66437 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=5f546745 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=742bd6ee snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Message handling failed in `eth`"       id=83ae5a6461390d373891ccebda8ba263f2d20e7981b157303fe0cef944fcacde conn=dyndial         err=EOF
t=2024-01-18T23:44:40-0500 lvl=info msg="Transaction pool stopped"
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=75760490 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=742bd6ee name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=83ed420d name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=1
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=eb89f99c name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=c0b37c61 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=1
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=7c6dee29 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=ca92bde8 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Removing Ethereum peer"                 peer=83ae5a64 snap=false
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=56c3c8b2 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=66dcc30d name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=ca0d7147 name=Geth/v1.2.5/linux-amd64/go1.20.5  connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=66361b67 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=5d528815 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=1
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=75c5d3bd name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=7786caab name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=ca7e690e name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=83ef8e00 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=1
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=56b66ced name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=6611eb66 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=ca63cb63 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=2a02c0d7 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=757bf730 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=77af1c55 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=82677fcd name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=83156fc1 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=cb0c4d75 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=2
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journaled generator progress"           progress=done
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=5619934d name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=663af41b name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=cd6eebe9 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=472acf2d name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=1
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=67b66ab0 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=ca6c4f08 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=8254eed2 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=75455dfb name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=578f3d0f name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=77a40944 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=5f247109 name=Geth/v1.2.5/linux-amd64/go1.20.5  connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=565b56ff name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=446bd23c name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=ca1da8cb name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=65cad417 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=83cf1330 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=5d4319fe name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=561011f4 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=5f3c26d8 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=746d159c name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=5689620a name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=56e986ac name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg=unregisterPeer                           peer=83ae5a64 name=erigon/v2.40.0-dev-3da15fcb/linux-amd64/go1.19.7 connectNum=0
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled disk layer"                  root=0x004203b0739afd18a9a94739300ae61c4a90831f7fa274d855a6f99eb04a053d
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xa94a8a24863cdb65a0682fae351607f92726ad3541ed922e4c3be21fa25ace19 parent=0x004203b0739afd18a9a94739300ae61c4a90831f7fa274d855a6f99eb04a053d
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x851377cdde0b567ebb240c5589f971629341f2b4f310f98f40f4beccab4eea76 parent=0xa94a8a24863cdb65a0682fae351607f92726ad3541ed922e4c3be21fa25ace19
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xe8a6b662f28945aa1f277ffafe8624f7cb8b4cf4404df1f6a29b0678bbb060fe parent=0x851377cdde0b567ebb240c5589f971629341f2b4f310f98f40f4beccab4eea76
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x5af98c8a479c1fec8a2ad363e5e78019e6aaada81a46db597efa04558e330a21 parent=0xe8a6b662f28945aa1f277ffafe8624f7cb8b4cf4404df1f6a29b0678bbb060fe
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x4c2015a5710f83ac1c204f514c9c574722227de8148d0fbf63c7b13e14054cfa parent=0x5af98c8a479c1fec8a2ad363e5e78019e6aaada81a46db597efa04558e330a21
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xb7d166ba1ed718ef544643c94602915e2de364520374ca0d0337a934eef58964 parent=0x4c2015a5710f83ac1c204f514c9c574722227de8148d0fbf63c7b13e14054cfa
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x795eb93fb08d9faec00908ffdf1d71cdb93dab3fee29b1cd93bc2012efabc88a parent=0xb7d166ba1ed718ef544643c94602915e2de364520374ca0d0337a934eef58964
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x4892b8da9657756ec47fed88f27a81ab756fe9bf2c203bfe97885ac8d34a3f99 parent=0x795eb93fb08d9faec00908ffdf1d71cdb93dab3fee29b1cd93bc2012efabc88a
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x9197a7b5b0f5038a5884fe81903e239c334a733672a3fa629c281103f09f4be1 parent=0x4892b8da9657756ec47fed88f27a81ab756fe9bf2c203bfe97885ac8d34a3f99
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xe88eadb28d42512b334eb8bac63609224153a2054b5db7efcb6b1359537a9906 parent=0x9197a7b5b0f5038a5884fe81903e239c334a733672a3fa629c281103f09f4be1
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x9bde97df2f73f95fd009ca5351e110cb2ffc6678edb3e0487066a45d179dea0d parent=0xe88eadb28d42512b334eb8bac63609224153a2054b5db7efcb6b1359537a9906
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x8aa72639a14f1fce70360f01318c4ea44366441775e7bd57eba4e2409ce5c511 parent=0x9bde97df2f73f95fd009ca5351e110cb2ffc6678edb3e0487066a45d179dea0d
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x43c35dec24037ff76370eca4dc1ac511d5be888859bb18b21c3d1cc36a93908f parent=0x8aa72639a14f1fce70360f01318c4ea44366441775e7bd57eba4e2409ce5c511
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x769b6d2299f85b71b50cb17c85d52434a71b31ae20166d2183eed3915feb8559 parent=0x43c35dec24037ff76370eca4dc1ac511d5be888859bb18b21c3d1cc36a93908f
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x2cf7705e924de55b6e3a7c02556c90481e5d49784336bcb86533ff44b6310488 parent=0x769b6d2299f85b71b50cb17c85d52434a71b31ae20166d2183eed3915feb8559
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x9e1d59ea82634db8a4baa63dbf350da0088026abeb1f5ad653bf298a2a125500 parent=0x2cf7705e924de55b6e3a7c02556c90481e5d49784336bcb86533ff44b6310488
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x85cadf617dbd4d43fcf6c2e86fcac0af57771818023774d2f3c9a8ed64cfafad parent=0x9e1d59ea82634db8a4baa63dbf350da0088026abeb1f5ad653bf298a2a125500
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x540f073a78e6b9f787a9ebfedb51a232d777a423d60d7a60dbd25ab346b51cca parent=0x85cadf617dbd4d43fcf6c2e86fcac0af57771818023774d2f3c9a8ed64cfafad
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x373ae32ae86bc61c4f351fb4ca97d29aefde25ff11cf1d6b641334a45f69dd08 parent=0x540f073a78e6b9f787a9ebfedb51a232d777a423d60d7a60dbd25ab346b51cca
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x70a5dfbc52ac0eb3e70795a4757e6c39ec1f3789fed7bcea36ea3eebb22e06d3 parent=0x373ae32ae86bc61c4f351fb4ca97d29aefde25ff11cf1d6b641334a45f69dd08
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x8ba30f480e0f13ecf0f9a1a7bb6bef1d869d592515dd4791ca820b0165967ff9 parent=0x70a5dfbc52ac0eb3e70795a4757e6c39ec1f3789fed7bcea36ea3eebb22e06d3
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x8d26ce7f0707ec4de4632173399e13a1fd330295ba45902b0f9ed54137e5bbd8 parent=0x8ba30f480e0f13ecf0f9a1a7bb6bef1d869d592515dd4791ca820b0165967ff9
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x9745e5e034f8ae4d7c722c0ab4060c54ee4546b8c1129545c9d9dd5cb7125650 parent=0x8d26ce7f0707ec4de4632173399e13a1fd330295ba45902b0f9ed54137e5bbd8
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xbbab1266213ba70579f801b0695551da052d7fce3a6a0e091ba06652a17d47f4 parent=0x9745e5e034f8ae4d7c722c0ab4060c54ee4546b8c1129545c9d9dd5cb7125650
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x0226074c07fb50e57f47cc5d07e8ab05f27cd79aa1be4110b1f0ad788641a673 parent=0xbbab1266213ba70579f801b0695551da052d7fce3a6a0e091ba06652a17d47f4
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xf272589158aeb10197168d2b3f578d2fe606cfe89ac9496e9ab443e7a9620b28 parent=0x0226074c07fb50e57f47cc5d07e8ab05f27cd79aa1be4110b1f0ad788641a673
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x59ad9f64d18e02333856cf9ff99a8daf8c8524eba295ce376bedb63f925c3629 parent=0xf272589158aeb10197168d2b3f578d2fe606cfe89ac9496e9ab443e7a9620b28
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xfb8b3ec31d850ed3c537608562209e918d2abcab77955f7803ffac62da98c004 parent=0x59ad9f64d18e02333856cf9ff99a8daf8c8524eba295ce376bedb63f925c3629
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x0278580c3ce9c603b16f00b7668eeb49665e27b3e647cdf6ad0a8d290d1c1490 parent=0xfb8b3ec31d850ed3c537608562209e918d2abcab77955f7803ffac62da98c004
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x5e812e9dfd5065150745a414b73678bcf53cbf9f0a3d97f22de4fa9e8b7a1ca8 parent=0x0278580c3ce9c603b16f00b7668eeb49665e27b3e647cdf6ad0a8d290d1c1490
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xab5e72d85ee8196954016d81fc0e4fc69b8516d7dd079f44bfc51c872bf066f5 parent=0x5e812e9dfd5065150745a414b73678bcf53cbf9f0a3d97f22de4fa9e8b7a1ca8
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x995f315449dd893089f67e8baa0794350077fd196108e92a77144cb5f77f78fc parent=0xab5e72d85ee8196954016d81fc0e4fc69b8516d7dd079f44bfc51c872bf066f5
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x590271cc1fe58990215053b62c8d4bc4919d4f451fdeb955a6c9714f0ca5a1eb parent=0x995f315449dd893089f67e8baa0794350077fd196108e92a77144cb5f77f78fc
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x09e5182dff4f6626c2cea62b0cc016e8d99270075a20de0f65712d8638294ff2 parent=0x590271cc1fe58990215053b62c8d4bc4919d4f451fdeb955a6c9714f0ca5a1eb
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x09e81daaf4ad05c1cdf11b4f28963460243e62e9fe83e95241d442374a5427a1 parent=0x09e5182dff4f6626c2cea62b0cc016e8d99270075a20de0f65712d8638294ff2
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x716655d28222963333430c4f34df2d82d0e3f2a98c977d7b145456bab18338e8 parent=0x09e81daaf4ad05c1cdf11b4f28963460243e62e9fe83e95241d442374a5427a1
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xe72fc25ba087f6147286fa5b41548d648873cdbff11b96638ea4ffb350e15280 parent=0x716655d28222963333430c4f34df2d82d0e3f2a98c977d7b145456bab18338e8
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x097a24ea2680b5007e6894f95b33aeb974378c1f933e3c4fb818a216e87ed948 parent=0xe72fc25ba087f6147286fa5b41548d648873cdbff11b96638ea4ffb350e15280
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xc79db2fd74b13056aacaf1665965410514cb26bc054baf1f222b27de9c48976d parent=0x097a24ea2680b5007e6894f95b33aeb974378c1f933e3c4fb818a216e87ed948
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x35e5070e95c80d128653f823551e7356ac428e6a7cecf7e840d1b660671a3045 parent=0xc79db2fd74b13056aacaf1665965410514cb26bc054baf1f222b27de9c48976d
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x82a7fb894b99dbc92932db1d9407691338b134770582fa93804d8c13b6849f94 parent=0x35e5070e95c80d128653f823551e7356ac428e6a7cecf7e840d1b660671a3045
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xc62554b4b7dab7e750fd7361ffd6d428c8eaf8560871a62fcc49545b0dcac420 parent=0x82a7fb894b99dbc92932db1d9407691338b134770582fa93804d8c13b6849f94
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x4a44b0b6a1557e07648911529b327cbaa687cd14592ee3bcd635d9fdfef7ceab parent=0xc62554b4b7dab7e750fd7361ffd6d428c8eaf8560871a62fcc49545b0dcac420
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x30646ed0c0d4a0908da372a713d125af7d88020e65ab6c958306a5b8f8b50456 parent=0x4a44b0b6a1557e07648911529b327cbaa687cd14592ee3bcd635d9fdfef7ceab
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xa4c3a1b9516ceaefdc082a110c6d4c396bd5058f98b245e0d47d261de99e32f7 parent=0x30646ed0c0d4a0908da372a713d125af7d88020e65ab6c958306a5b8f8b50456
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xe94c8781b0e4f2a993e8a75200d8c6e00856b06866e8c4618263efed6ee5c59b parent=0xa4c3a1b9516ceaefdc082a110c6d4c396bd5058f98b245e0d47d261de99e32f7
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x4add20937c1416cba1100d2d28ac5b6c7df969df8fa1277e6ecb9535eae3053f parent=0xe94c8781b0e4f2a993e8a75200d8c6e00856b06866e8c4618263efed6ee5c59b
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x4c09105f2a21b0ca633806b2c9d23c0a74f58fcebbd61ef5cd4955d8375cf54b parent=0x4add20937c1416cba1100d2d28ac5b6c7df969df8fa1277e6ecb9535eae3053f
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x8e60f4e9f3a77296f3bac22e43dc4a2d96c7d76b714f4b5a41b59f6ad5621291 parent=0x4c09105f2a21b0ca633806b2c9d23c0a74f58fcebbd61ef5cd4955d8375cf54b
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xa62ca88406fc2014d2fb6093fa3249b347fd52825e0f99a1cfc0d5b929e10359 parent=0x8e60f4e9f3a77296f3bac22e43dc4a2d96c7d76b714f4b5a41b59f6ad5621291
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x813f9ce17f18c5adede48cb1511df8c0a7d347d8173ddd4d8f04e0cbcd02d0d8 parent=0xa62ca88406fc2014d2fb6093fa3249b347fd52825e0f99a1cfc0d5b929e10359
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x9b08e1aea4d3ac62232d37c4caaad68a5881204e202802676e392f075f8c0c0b parent=0x813f9ce17f18c5adede48cb1511df8c0a7d347d8173ddd4d8f04e0cbcd02d0d8
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x6e5b276bbce21728addd6c31c90788f5ecde9e13647ea394d8512bf419e37e8e parent=0x9b08e1aea4d3ac62232d37c4caaad68a5881204e202802676e392f075f8c0c0b
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x50ef65477d1733933ad384c3ea23b7334146e6813c76a8fa007ea49c84ec6669 parent=0x6e5b276bbce21728addd6c31c90788f5ecde9e13647ea394d8512bf419e37e8e
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x8e2898708858b66154e55174a0d42de72c5c2ce626cbf4c99f114bfe634d934a parent=0x50ef65477d1733933ad384c3ea23b7334146e6813c76a8fa007ea49c84ec6669
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x9d7229f05b3759934794712717f372b09bba6ca39cbea84665cb80733e49b65a parent=0x8e2898708858b66154e55174a0d42de72c5c2ce626cbf4c99f114bfe634d934a
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xfd613cdc97335057c42220bbaae22e33319a8da860e94bef8e099cfb7ffa6464 parent=0x9d7229f05b3759934794712717f372b09bba6ca39cbea84665cb80733e49b65a
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x78b2ca60e707d49f5c730f10c03a9b54a1de43c445be0738972eb4cd9a9483a3 parent=0xfd613cdc97335057c42220bbaae22e33319a8da860e94bef8e099cfb7ffa6464
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xa269241d720e093c5e13e521f1c4d07c947ff3e67e0b79aec65db89bdaff17ab parent=0x78b2ca60e707d49f5c730f10c03a9b54a1de43c445be0738972eb4cd9a9483a3
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xeeafa82773f2483cd5c7c878b97daa89d27ab9712a447972914b8e1828b58be0 parent=0xa269241d720e093c5e13e521f1c4d07c947ff3e67e0b79aec65db89bdaff17ab
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x1b77fdb76d42be77e818fa361c876179246de972fce5dec307603b9fac10a6a8 parent=0xeeafa82773f2483cd5c7c878b97daa89d27ab9712a447972914b8e1828b58be0
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xb704a9dc011f060d6f06b0b01845aa4590698ffceb8ec8b564d2256cfd1c0528 parent=0x1b77fdb76d42be77e818fa361c876179246de972fce5dec307603b9fac10a6a8
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x4d8c0eb03cf7a5005369daa55bf13f090e7e6881aa3fa093e05d01b54b5616fe parent=0xb704a9dc011f060d6f06b0b01845aa4590698ffceb8ec8b564d2256cfd1c0528
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xe88f32a63db0db10e09c5427dd59c029c80699d1ac745f4a070910fb1daa65dd parent=0x4d8c0eb03cf7a5005369daa55bf13f090e7e6881aa3fa093e05d01b54b5616fe
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xfbf90d5bba10c066ba9beb691c699d156125dd8a66dab266052046348638fb24 parent=0xe88f32a63db0db10e09c5427dd59c029c80699d1ac745f4a070910fb1daa65dd
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x59d0aa40f401233d8d94639ae7116dea9d3c0eff9a1034b88ef312a462ad0a96 parent=0xfbf90d5bba10c066ba9beb691c699d156125dd8a66dab266052046348638fb24
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x54fbfcb7087f958339c3c19bb5835e967f61f3a21e9a8e316cb7abd490d29cf2 parent=0x59d0aa40f401233d8d94639ae7116dea9d3c0eff9a1034b88ef312a462ad0a96
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x0844f58169191171d1883939aee4ea8b2ba7437e4dafd7bced396c6a3b420df3 parent=0x54fbfcb7087f958339c3c19bb5835e967f61f3a21e9a8e316cb7abd490d29cf2
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xced071d47188b8bfa3a893daa829a5cf7b3437032edf8e75bcad341f1d51454e parent=0x0844f58169191171d1883939aee4ea8b2ba7437e4dafd7bced396c6a3b420df3
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x8976186aaa6144e37e52ef475d1efcc0c82cf3335ae189ffb72eb71ba3ba8a45 parent=0xced071d47188b8bfa3a893daa829a5cf7b3437032edf8e75bcad341f1d51454e
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xfa54a0bc04314fada970fc3691f71c648cea3c0ac3ce2402e70dab28451f2f27 parent=0x8976186aaa6144e37e52ef475d1efcc0c82cf3335ae189ffb72eb71ba3ba8a45
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x3586bcf22e171be833fde30c70cbd325b9c75bc01cc72014eaf6ff3bc42bf5aa parent=0xfa54a0bc04314fada970fc3691f71c648cea3c0ac3ce2402e70dab28451f2f27
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xe9415338201a1be2ca2aaabd8a989c0245909505803325bf7b2b008e43835019 parent=0x3586bcf22e171be833fde30c70cbd325b9c75bc01cc72014eaf6ff3bc42bf5aa
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x6e435ea0112e28bf5b44cb6f4a266a3bc48895683724dc3a0e6bf75a645b5b04 parent=0xe9415338201a1be2ca2aaabd8a989c0245909505803325bf7b2b008e43835019
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x82c87f226a72419a7477adc9e9002ecd3ddd0270851385892488e2574c1e85b3 parent=0x6e435ea0112e28bf5b44cb6f4a266a3bc48895683724dc3a0e6bf75a645b5b04
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xec4ee7faa4dd9cb77f0213a240606a98ecff056519f6706e27ff02186eeecab7 parent=0x82c87f226a72419a7477adc9e9002ecd3ddd0270851385892488e2574c1e85b3
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x5b860ca428f09ed7d13c5c344b7b1241c8196c0808eda13e575138b9654e8cf2 parent=0xec4ee7faa4dd9cb77f0213a240606a98ecff056519f6706e27ff02186eeecab7
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x6f8c48bb5ebc761f42c2f4018ff52b382e386c94ce9bf7afd701e41b320f7d41 parent=0x5b860ca428f09ed7d13c5c344b7b1241c8196c0808eda13e575138b9654e8cf2
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xf7e65e7b659e3003ca320903ed0dabb257002f2e82b4862745fee59c1136aa1f parent=0x6f8c48bb5ebc761f42c2f4018ff52b382e386c94ce9bf7afd701e41b320f7d41
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xfe6c792f5ba19907737837d26d90781b8682d26ad9e4c50139e3abd02ea3c703 parent=0xf7e65e7b659e3003ca320903ed0dabb257002f2e82b4862745fee59c1136aa1f
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x0ca66bd54ff97be1ffb54cb0347ad03bb7d672dea16ab4a66bd5f7ad40fcdc63 parent=0xfe6c792f5ba19907737837d26d90781b8682d26ad9e4c50139e3abd02ea3c703
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xd5ba20379ac48f24d10ff1848965d1911ad2a92163a16fb49c3b705b57b6d268 parent=0x0ca66bd54ff97be1ffb54cb0347ad03bb7d672dea16ab4a66bd5f7ad40fcdc63
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x288a7dde55f3006b2a87ed1b0f43ade59eaf385371ebd1847eccdd10db2d1504 parent=0xd5ba20379ac48f24d10ff1848965d1911ad2a92163a16fb49c3b705b57b6d268
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x7e81fc54df71830edc55a52b0c7976bdf8d54d2d47b54979c50f0bfc96a68067 parent=0x288a7dde55f3006b2a87ed1b0f43ade59eaf385371ebd1847eccdd10db2d1504
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x632ac04dd90fc055b0acd3d5a6257a97e269502366223b3d67f1c2b5e0efcf4f parent=0x7e81fc54df71830edc55a52b0c7976bdf8d54d2d47b54979c50f0bfc96a68067
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x214d0c1be8a9bd0eb79aaf455bc617a672e345be8510bf7f91cbbcf0e5cc8967 parent=0x632ac04dd90fc055b0acd3d5a6257a97e269502366223b3d67f1c2b5e0efcf4f
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xb6e5daccdbaf2cb66653db2ef3cda5bba17ac08503c7c6e97de8babfbbccfa19 parent=0x214d0c1be8a9bd0eb79aaf455bc617a672e345be8510bf7f91cbbcf0e5cc8967
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xf6a096ba254ecc96fa14e5482c565b87894cbe498ed67585e898dc354978a13e parent=0xb6e5daccdbaf2cb66653db2ef3cda5bba17ac08503c7c6e97de8babfbbccfa19
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x9f1eeffda133e2fa88f5fbd2e171e9d1ea2ce12ea275c9f2cae8fcd9adb8c8a1 parent=0xf6a096ba254ecc96fa14e5482c565b87894cbe498ed67585e898dc354978a13e
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xe13e3de6b37e08792d55e36fd458a677b717d5bdddba033f213abb5431dcdece parent=0x9f1eeffda133e2fa88f5fbd2e171e9d1ea2ce12ea275c9f2cae8fcd9adb8c8a1
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x82e0a89519f0648dd730815f9bf7701811d3ce928670b11a072a7019b5fb2217 parent=0xe13e3de6b37e08792d55e36fd458a677b717d5bdddba033f213abb5431dcdece
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x5972958a9b77cf3dcfef460beca82d72b4e133e6c569a2c268162f1661597808 parent=0x82e0a89519f0648dd730815f9bf7701811d3ce928670b11a072a7019b5fb2217
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x1b4dd072fd8d0a18dd6eaf1f24572cf7384efcf0be33b162b70543a063a83d5f parent=0x5972958a9b77cf3dcfef460beca82d72b4e133e6c569a2c268162f1661597808
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x9cd8cf86cfdfe62934467f009c31ef34e1b7bc6f0551856ad4da51bff4339025 parent=0x1b4dd072fd8d0a18dd6eaf1f24572cf7384efcf0be33b162b70543a063a83d5f
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x45f717afe426de40949ba7e9e6ab19814e1744cf220bb94bdfbce03b2434a7c8 parent=0x9cd8cf86cfdfe62934467f009c31ef34e1b7bc6f0551856ad4da51bff4339025
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xbea6c884f70ef6804872d9b608189092d3140358a70eb29314cf87adf6ccbd1e parent=0x45f717afe426de40949ba7e9e6ab19814e1744cf220bb94bdfbce03b2434a7c8
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x83470f36b319d8419402a2c23ecfc8cc330217c3b21c4b2093b68b34271042eb parent=0xbea6c884f70ef6804872d9b608189092d3140358a70eb29314cf87adf6ccbd1e
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xea160cb6b1117c6fb555640992582a9d49cccf54b8f614044d76d823f719daf9 parent=0x83470f36b319d8419402a2c23ecfc8cc330217c3b21c4b2093b68b34271042eb
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x7317e9605aa8dcbdeff6153dc4bcc4a4706ae756712b744ebd6e23796ab8666e parent=0xea160cb6b1117c6fb555640992582a9d49cccf54b8f614044d76d823f719daf9
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x597dd0d177480d647c76c04d84c00b909e9dc495f21adbe3aa8631b487b11453 parent=0x7317e9605aa8dcbdeff6153dc4bcc4a4706ae756712b744ebd6e23796ab8666e
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xc93b101daac8b521e7b7e7d150cf5aa192f4661b55cdf38cef2dff1bd1502931 parent=0x597dd0d177480d647c76c04d84c00b909e9dc495f21adbe3aa8631b487b11453
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x45cf4ab106241bea8f0e6622758bb1f4d2ef4cb7424769b2dc95c76106ba393d parent=0xc93b101daac8b521e7b7e7d150cf5aa192f4661b55cdf38cef2dff1bd1502931
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x4f740f68cf7854b2ad441b4c1792db8d943ca2e7dd151ec6c1d290ac9589c92e parent=0x45cf4ab106241bea8f0e6622758bb1f4d2ef4cb7424769b2dc95c76106ba393d
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x6b2f9f35df6e90f95bad7b6966a7451989cc6ba2d6aaa9c43f1fa28430980333 parent=0x4f740f68cf7854b2ad441b4c1792db8d943ca2e7dd151ec6c1d290ac9589c92e
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x0142e7932de50c23aead7bca193e08a91c2d2630a1062ed256433a533da9f6f9 parent=0x6b2f9f35df6e90f95bad7b6966a7451989cc6ba2d6aaa9c43f1fa28430980333
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xbb0bc4494aea8fc732422d273af877e9897ba35cd400eb616588be020e330446 parent=0x0142e7932de50c23aead7bca193e08a91c2d2630a1062ed256433a533da9f6f9
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x89871d58ed1b7b45dacf4dc3765bb9a2fe083d34c8aec1fb804790f6617e3bc5 parent=0xbb0bc4494aea8fc732422d273af877e9897ba35cd400eb616588be020e330446
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xc36e71b3ce1349991e5c326d8e30ece1e5dfbbee2839f1293398774124a603cf parent=0x89871d58ed1b7b45dacf4dc3765bb9a2fe083d34c8aec1fb804790f6617e3bc5
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x069ef32958c41f86c3adc736893274b633e08fb4b94051137a4153b3ea9623d7 parent=0xc36e71b3ce1349991e5c326d8e30ece1e5dfbbee2839f1293398774124a603cf
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xa280d6db2464b6b81db7ac1f6f119b78ff8d4af43c1a267cad09af3ae623a57c parent=0x069ef32958c41f86c3adc736893274b633e08fb4b94051137a4153b3ea9623d7
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xf51d14750f5b82da47d2a5306407ca903950b151fafb7fb250846cee510445f0 parent=0xa280d6db2464b6b81db7ac1f6f119b78ff8d4af43c1a267cad09af3ae623a57c
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xcd7528dd4c1a841071846bd5f75d0ca6e29cc7faacd0a5dde07e93ce6373f022 parent=0xf51d14750f5b82da47d2a5306407ca903950b151fafb7fb250846cee510445f0
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xe5d591563f818b40d16129a3cea6939447b30c65a4fa847b7a3baec6ce7446df parent=0xcd7528dd4c1a841071846bd5f75d0ca6e29cc7faacd0a5dde07e93ce6373f022
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x3943c2b3ecc80e075cc4b3c8660059cd1bc399134220a5aae40909a499e8d646 parent=0xe5d591563f818b40d16129a3cea6939447b30c65a4fa847b7a3baec6ce7446df
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xc1880fc2f89ed33ef8804e1575930fb7ecf9aad2c3ff14b9b1536c327aac3e5f parent=0x3943c2b3ecc80e075cc4b3c8660059cd1bc399134220a5aae40909a499e8d646
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x7d4c3289235f626a056cacde2858e13250b674682ae4f5b2adfb9e0377418d51 parent=0xc1880fc2f89ed33ef8804e1575930fb7ecf9aad2c3ff14b9b1536c327aac3e5f
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x3ade81b8117785da2e2e105e0c421be496b1f454649ae09cd5a5b69e0552875f parent=0x7d4c3289235f626a056cacde2858e13250b674682ae4f5b2adfb9e0377418d51
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x4058525a3a4887118cbac772c9997061bccbc63b8558e38534eecacf2c75064e parent=0x3ade81b8117785da2e2e105e0c421be496b1f454649ae09cd5a5b69e0552875f
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xba27ece74200fc3af8c8a17a2c5358176e7d389263d7e06d5f0bfd7baf9af60e parent=0x4058525a3a4887118cbac772c9997061bccbc63b8558e38534eecacf2c75064e
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x020dde88451ddfd83e15c9db07d158c18a956fbd4d7bba4e9c0973fc7fdb23fd parent=0xba27ece74200fc3af8c8a17a2c5358176e7d389263d7e06d5f0bfd7baf9af60e
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x369ec9bc91400f984eafe4169792ba03105bdac2e0c2ee187c301baf512ccfff parent=0x020dde88451ddfd83e15c9db07d158c18a956fbd4d7bba4e9c0973fc7fdb23fd
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xe2965fa6cd76b1231d87ff24adef112f35f3f0b4663b13cabb48051e85868e6a parent=0x369ec9bc91400f984eafe4169792ba03105bdac2e0c2ee187c301baf512ccfff
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x8f6c5d6eb659f18a2cdd71c627108da10e003ccc81fb135f628e1e4901506666 parent=0xe2965fa6cd76b1231d87ff24adef112f35f3f0b4663b13cabb48051e85868e6a
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xb11a4c36a6a17453543f64282270bda0302aa97c62a8857952bd002f2e869c96 parent=0x8f6c5d6eb659f18a2cdd71c627108da10e003ccc81fb135f628e1e4901506666
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xc97a6e134330de58517aca9e16309e3147ab46b9cc0e361b99e9d39cddd50754 parent=0xb11a4c36a6a17453543f64282270bda0302aa97c62a8857952bd002f2e869c96
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xdb402e5d63d474d90e253e230a042791159e864dc8e79fffbfe0e3f540d2cb0d parent=0xc97a6e134330de58517aca9e16309e3147ab46b9cc0e361b99e9d39cddd50754
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x198eba6081011d63f61ce792083412e68d0433feef9d5023404a02b09619988b parent=0xdb402e5d63d474d90e253e230a042791159e864dc8e79fffbfe0e3f540d2cb0d
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0xe0956d05d6b424a2e4efb5ef799768700563b26fac9b7c3e81adadfcbb729bd7 parent=0x198eba6081011d63f61ce792083412e68d0433feef9d5023404a02b09619988b
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Journalled diff layer"                  root=0x2da127f53adf9ed77f3cb2ba744915df8a6410296d45ddd963141cf81148f417 parent=0xe0956d05d6b424a2e4efb5ef799768700563b26fac9b7c3e81adadfcbb729bd7
t=2024-01-18T23:44:40-0500 lvl=info msg="Writing cached state to disk"           block=35,218,824 hash=0x9410b2b38a0a35de84d5107621b0b99632845cf8a204c8316e6814310f0bbd25 root=0x2da127f53adf9ed77f3cb2ba744915df8a6410296d45ddd963141cf81148f417
t=2024-01-18T23:44:40-0500 lvl=info msg="Persisted trie from memory database"    nodes=0 size="0.00 B"     time="41.66µs"  gcnodes=0 gcsize="0.00 B" gctime="462.569µs" livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=info msg="Writing cached state to disk"           block=35,218,823 hash=0xcc0faef43a590038ab99a669386a06f9bf397528750f95a385e48e92797cec12 root=0xe0956d05d6b424a2e4efb5ef799768700563b26fac9b7c3e81adadfcbb729bd7
t=2024-01-18T23:44:40-0500 lvl=info msg="Persisted trie from memory database"    nodes=0 size="0.00 B"     time="1.42µs"   gcnodes=0 gcsize="0.00 B" gctime=0s          livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=info msg="Writing cached state to disk"           block=35,218,697 hash=0xa778549eac32fa7d2ca962eebc9555b0b98419c3f3cbdfe0ee07e7886c2f5c0c root=0x851377cdde0b567ebb240c5589f971629341f2b4f310f98f40f4beccab4eea76
t=2024-01-18T23:44:40-0500 lvl=info msg="Persisted trie from memory database"    nodes=0 size="0.00 B"     time="1.24µs"   gcnodes=0 gcsize="0.00 B" gctime=0s          livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=info msg="Writing snapshot state to disk"         root=0x004203b0739afd18a9a94739300ae61c4a90831f7fa274d855a6f99eb04a053d
t=2024-01-18T23:44:40-0500 lvl=info msg="Persisted trie from memory database"    nodes=0 size="0.00 B"     time=740ns      gcnodes=0 gcsize="0.00 B" gctime=0s          livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=info msg="Writing snapshot state to disk"         root=0x004203b0739afd18a9a94739300ae61c4a90831f7fa274d855a6f99eb04a053d
t=2024-01-18T23:44:40-0500 lvl=info msg="Persisted trie from memory database"    nodes=0 size="0.00 B"     time=880ns      gcnodes=0 gcsize="0.00 B" gctime=0s          livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time="1.16µs"   gcnodes=0 gcsize="0.00 B" gctime=610ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=150ns      gcnodes=0 gcsize="0.00 B" gctime=650ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime=690ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime=720ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime=750ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime=790ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime=830ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime=870ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime=900ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime=940ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=130ns      gcnodes=0 gcsize="0.00 B" gctime=980ns       livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.02µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.06µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.1µs"     livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.14µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.18µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.21µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.24µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.28µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="1.31µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.35µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.38µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="1.41µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=280ns      gcnodes=0 gcsize="0.00 B" gctime="1.49µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=200ns      gcnodes=0 gcsize="0.00 B" gctime="1.55µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.58µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.62µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.66µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.7µs"     livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="1.73µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.76µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.79µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.83µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.87µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="1.9µs"     livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.94µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="1.98µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.01µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.05µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.09µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.13µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.17µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.21µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.25µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.29µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="2.32µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.36µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.39µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.43µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.47µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.51µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=280ns      gcnodes=0 gcsize="0.00 B" gctime="2.61µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=280ns      gcnodes=0 gcsize="0.00 B" gctime="2.7µs"     livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.74µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=250ns      gcnodes=0 gcsize="0.00 B" gctime="2.81µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="2.84µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.88µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.92µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="2.96µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="2.99µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=250ns      gcnodes=0 gcsize="0.00 B" gctime="3.06µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="3.1µs"     livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="3.13µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="3.17µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="3.21µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=110ns      gcnodes=0 gcsize="0.00 B" gctime="3.25µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="3.279µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="3.319µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="3.35µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=309ns      gcnodes=0 gcsize="0.00 B" gctime="3.459µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=120ns      gcnodes=0 gcsize="0.00 B" gctime="3.51µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="3.541µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="3.581µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=120ns      gcnodes=0 gcsize="0.00 B" gctime="3.621µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="3.652µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="3.692µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="3.721µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=249ns      gcnodes=0 gcsize="0.00 B" gctime="3.79µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=120ns      gcnodes=0 gcsize="0.00 B" gctime="3.83µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="3.859µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="3.89µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="3.921µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="3.961µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="4.001µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="4.032µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="4.063µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="4.103µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="4.134µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="4.165µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="4.205µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="4.245µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="4.285µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="4.316µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="4.356µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="4.387µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="4.418µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="4.458µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="4.489µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="4.52µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="4.551µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="4.591µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="4.631µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="4.662µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="4.693µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=300ns      gcnodes=0 gcsize="0.00 B" gctime="4.762µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="4.802µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="4.842µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=120ns      gcnodes=0 gcsize="0.00 B" gctime="4.882µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="4.911µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="4.942µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="4.982µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="5.022µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="5.062µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="5.102µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="5.142µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="5.182µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="5.222µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=300ns      gcnodes=0 gcsize="0.00 B" gctime="5.302µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="5.331µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="5.362µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="5.391µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=109ns      gcnodes=0 gcsize="0.00 B" gctime="5.431µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="5.471µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=260ns      gcnodes=0 gcsize="0.00 B" gctime="5.551µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="5.591µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=111ns      gcnodes=0 gcsize="0.00 B" gctime="5.631µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="5.66µs"    livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=dbug msg="Dereferenced trie from memory database" nodes=0 size="0.00 B"     time=100ns      gcnodes=0 gcsize="0.00 B" gctime="5.691µs"   livenodes=0 livesize="0.00 B"
t=2024-01-18T23:44:40-0500 lvl=info msg="Blockchain stopped"
t=2024-01-18T23:44:41-0500 lvl=dbug msg="Read error"                             err="accept tcp [::]:30311: use of closed network connection"
t=2024-01-18T23:44:41-0500 lvl=dbug msg="UDP read error"                         err="read udp [::]:30311: use of closed network connection"
t=2024-01-18T23:44:46-0500 lvl=warn msg="stop p2p server timeout, forcing stop"

Then when you start the node again (after systemctl start of the service), the unrecoverable state:

t=2024-01-18T23:46:03-0500 lvl=dbug msg="FS scan times" list=2.122438ms set="1.62µs" diff="4.98µs"
t=2024-01-18T23:46:03-0500 lvl=dbug msg="Sanitizing Go's GC trigger" percent=25
t=2024-01-18T23:46:03-0500 lvl=info msg="Using leveldb as the backing database"
t=2024-01-18T23:46:03-0500 lvl=info msg="Allocated cache and file handles"      database=/x/binance-smart-chain/bsc-geth/geth/chaindata cache="1.60 GiB" handles=262,144 readonly=true
t=2024-01-18T23:46:08-0500 lvl=info msg="Using LevelDB as the backing database"
t=2024-01-18T23:46:08-0500 lvl=dbug msg="FS scan times" list="31.78µs" set="1.62µs" diff="10.091µs"
t=2024-01-18T23:46:08-0500 lvl=dbug msg="Sanitizing Go's GC trigger" percent=25
t=2024-01-18T23:46:09-0500 lvl=info msg="Using leveldb as the backing database"
t=2024-01-18T23:46:09-0500 lvl=info msg="Allocated cache and file handles"      database=/x/binance-smart-chain/bsc-geth/geth/chaindata cache="1.60 GiB" handles=262,144 readonly=true
t=2024-01-18T23:46:12-0500 lvl=info msg="Using LevelDB as the backing database"
t=2024-01-18T23:46:13-0500 lvl=dbug msg="FS scan times" list="99.121µs" set=910ns diff="1.67µs"
t=2024-01-18T23:46:13-0500 lvl=dbug msg="Sanitizing Go's GC trigger" percent=25
t=2024-01-18T23:46:13-0500 lvl=info msg="Using leveldb as the backing database"
t=2024-01-18T23:46:13-0500 lvl=info msg="Allocated cache and file handles"      database=/x/binance-smart-chain/bsc-geth/geth/chaindata cache="1.60 GiB" handles=262,144 readonly=true
t=2024-01-18T23:46:17-0500 lvl=info msg="Using LevelDB as the backing database"
t=2024-01-18T23:46:17-0500 lvl=dbug msg="FS scan times" list="34.441µs" set=469ns diff="2.2µs"
t=2024-01-18T23:46:17-0500 lvl=dbug msg="FS scan times" list="13.069µs" set=411ns diff=420ns
t=2024-01-18T23:46:17-0500 lvl=dbug msg="Sanitizing Go's GC trigger" percent=25
t=2024-01-18T23:46:18-0500 lvl=info msg="Using leveldb as the backing database"
t=2024-01-18T23:46:18-0500 lvl=info msg="Allocated cache and file handles"      database=/xbinance-smart-chain/bsc-geth/geth/chaindata cache="1.60 GiB" handles=262,144 readonly=true
t=2024-01-18T23:46:21-0500 lvl=info msg="Using LevelDB as the backing database"

From journalctl logs:

Jan 19 00:11:21 BSC-VM-01 sh[113208]: INFO [01-19|00:11:21.232] Starting Geth on BSC mainnet...
Jan 19 00:11:21 BSC-VM-01 sh[113208]: INFO [01-19|00:11:21.232] Bumping default cache on mainnet         provided=1024 updated=4096
Jan 19 00:11:21 BSC-VM-01 sh[113208]: WARN [01-19|00:11:21.233] Config field is deprecated and won't have an effect name=ethconfig.Config.EWASMInterpreter
Jan 19 00:11:21 BSC-VM-01 sh[113208]: WARN [01-19|00:11:21.233] Config field is deprecated and won't have an effect name=ethconfig.Config.EVMInterpreter
Jan 19 00:11:21 BSC-VM-01 sh[113208]: INFO [01-19|00:11:21.235] Maximum peer count                       ETH=150 LES=0 total=150
Jan 19 00:11:21 BSC-VM-01 sh[113208]: INFO [01-19|00:11:21.236] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
Jan 19 00:11:25 BSC-VM-01 sh[113208]: Chain metadata
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   databaseVersion: 8 (0x8)
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   headBlockHash: 0x9410b2b38a0a35de84d5107621b0b99632845cf8a204c8316e6814310f0bbd25
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   headFastBlockHash: 0x9410b2b38a0a35de84d5107621b0b99632845cf8a204c8316e6814310f0bbd25
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   headHeaderHash: 0x9410b2b38a0a35de84d5107621b0b99632845cf8a204c8316e6814310f0bbd25
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   lastPivotNumber: 6920578 (0x699982)
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   len(snapshotSyncStatus): 0 bytes
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   snapshotDisabled: false
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   snapshotJournal: 4091061 bytes
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   snapshotRecoveryNumber: 33908906 (0x20568aa)
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   snapshotRoot: 0x004203b0739afd18a9a94739300ae61c4a90831f7fa274d855a6f99eb04a053d
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   txIndexTail: 0 (0x0)
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   fastTxLookupLimit: 2350000 (0x23dbb0)
Jan 19 00:11:25 BSC-VM-01 sh[113208]: Fatal: Could not open database: open /x/binance-smart-chain/bsc-geth/geth/chaindata/ancient/chain/headers.cidx: no such file or directory

The same was mentioned in #2110, #2049, #1997, #2018

@zzzckck
Copy link
Collaborator

zzzckck commented Jan 22, 2024

I was able to reproduce the same issue, the node was stuck (#2145, #2104) based on the recommendation to remove the "nodekey" file from the downloaded snapshot or restarting the node - the db got fried, but I was able to capture the logs:

After systemctl stop of the service

Logs
Then when you start the node again (after systemctl start of the service), the unrecoverable state:

t=2024-01-18T23:46:03-0500 lvl=dbug msg="FS scan times" list=2.122438ms set="1.62µs" diff="4.98µs"
t=2024-01-18T23:46:03-0500 lvl=dbug msg="Sanitizing Go's GC trigger" percent=25
t=2024-01-18T23:46:03-0500 lvl=info msg="Using leveldb as the backing database"
t=2024-01-18T23:46:03-0500 lvl=info msg="Allocated cache and file handles"      database=/x/binance-smart-chain/bsc-geth/geth/chaindata cache="1.60 GiB" handles=262,144 readonly=true
t=2024-01-18T23:46:08-0500 lvl=info msg="Using LevelDB as the backing database"
t=2024-01-18T23:46:08-0500 lvl=dbug msg="FS scan times" list="31.78µs" set="1.62µs" diff="10.091µs"
t=2024-01-18T23:46:08-0500 lvl=dbug msg="Sanitizing Go's GC trigger" percent=25
t=2024-01-18T23:46:09-0500 lvl=info msg="Using leveldb as the backing database"
t=2024-01-18T23:46:09-0500 lvl=info msg="Allocated cache and file handles"      database=/x/binance-smart-chain/bsc-geth/geth/chaindata cache="1.60 GiB" handles=262,144 readonly=true
t=2024-01-18T23:46:12-0500 lvl=info msg="Using LevelDB as the backing database"
t=2024-01-18T23:46:13-0500 lvl=dbug msg="FS scan times" list="99.121µs" set=910ns diff="1.67µs"
t=2024-01-18T23:46:13-0500 lvl=dbug msg="Sanitizing Go's GC trigger" percent=25
t=2024-01-18T23:46:13-0500 lvl=info msg="Using leveldb as the backing database"
t=2024-01-18T23:46:13-0500 lvl=info msg="Allocated cache and file handles"      database=/x/binance-smart-chain/bsc-geth/geth/chaindata cache="1.60 GiB" handles=262,144 readonly=true
t=2024-01-18T23:46:17-0500 lvl=info msg="Using LevelDB as the backing database"
t=2024-01-18T23:46:17-0500 lvl=dbug msg="FS scan times" list="34.441µs" set=469ns diff="2.2µs"
t=2024-01-18T23:46:17-0500 lvl=dbug msg="FS scan times" list="13.069µs" set=411ns diff=420ns
t=2024-01-18T23:46:17-0500 lvl=dbug msg="Sanitizing Go's GC trigger" percent=25
t=2024-01-18T23:46:18-0500 lvl=info msg="Using leveldb as the backing database"
t=2024-01-18T23:46:18-0500 lvl=info msg="Allocated cache and file handles"      database=/xbinance-smart-chain/bsc-geth/geth/chaindata cache="1.60 GiB" handles=262,144 readonly=true
t=2024-01-18T23:46:21-0500 lvl=info msg="Using LevelDB as the backing database"

From journalctl logs:

Jan 19 00:11:21 BSC-VM-01 sh[113208]: INFO [01-19|00:11:21.232] Starting Geth on BSC mainnet...
Jan 19 00:11:21 BSC-VM-01 sh[113208]: INFO [01-19|00:11:21.232] Bumping default cache on mainnet         provided=1024 updated=4096
Jan 19 00:11:21 BSC-VM-01 sh[113208]: WARN [01-19|00:11:21.233] Config field is deprecated and won't have an effect name=ethconfig.Config.EWASMInterpreter
Jan 19 00:11:21 BSC-VM-01 sh[113208]: WARN [01-19|00:11:21.233] Config field is deprecated and won't have an effect name=ethconfig.Config.EVMInterpreter
Jan 19 00:11:21 BSC-VM-01 sh[113208]: INFO [01-19|00:11:21.235] Maximum peer count                       ETH=150 LES=0 total=150
Jan 19 00:11:21 BSC-VM-01 sh[113208]: INFO [01-19|00:11:21.236] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
Jan 19 00:11:25 BSC-VM-01 sh[113208]: Chain metadata
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   databaseVersion: 8 (0x8)
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   headBlockHash: 0x9410b2b38a0a35de84d5107621b0b99632845cf8a204c8316e6814310f0bbd25
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   headFastBlockHash: 0x9410b2b38a0a35de84d5107621b0b99632845cf8a204c8316e6814310f0bbd25
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   headHeaderHash: 0x9410b2b38a0a35de84d5107621b0b99632845cf8a204c8316e6814310f0bbd25
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   lastPivotNumber: 6920578 (0x699982)
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   len(snapshotSyncStatus): 0 bytes
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   snapshotDisabled: false
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   snapshotJournal: 4091061 bytes
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   snapshotRecoveryNumber: 33908906 (0x20568aa)
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   snapshotRoot: 0x004203b0739afd18a9a94739300ae61c4a90831f7fa274d855a6f99eb04a053d
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   txIndexTail: 0 (0x0)
Jan 19 00:11:25 BSC-VM-01 sh[113208]:   fastTxLookupLimit: 2350000 (0x23dbb0)
Jan 19 00:11:25 BSC-VM-01 sh[113208]: Fatal: Could not open database: open /x/binance-smart-chain/bsc-geth/geth/chaindata/ancient/chain/headers.cidx: no such file or directory

The same was mentioned in #2110, #2049, #1997, #2018

This issue has been resolved by this PR: #2155
Will be included in next release, v1.3.8 target date around end of Jan

@zzzckck zzzckck closed this as completed Jan 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@zzzckck zzzckck

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@deepcrazy @HarshitJoshi @zzzckck