Clique doesn't seal any blocks

[0x366]

System information

OS & Version: OSX
Commit hash : c170fa2

genesis.json:

{
    "config": {
        "chainId": 1,
        "homesteadBlock": 0,
        "eip150Block": 0,
        "eip155Block": 0,
        "eip158Block": 0,
        "byzantiumBlock": 0,
        "constantinopleBlock": 0,
        "petersburgBlock": 0,
        "istanbulBlock": 0,
        "muirGlacierBlock": 0,
        "berlinBlock": 0,
        "londonBlock": 0,
        "arrowGlacierBlock": 0,
        "grayGlacierBlock": 0,
        "terminalTotalDifficultyPassed": true,
        "shanghaiTime": 0,
        "clique": {
            "period": 2,
            "epoch": 1000
        }
    },
    "timestamp": "0x65dd9a5f",
    "difficulty": "1",
    "gasLimit": "30000000",
    "extradata": "0x00000000000000000000000000000000000000000000000000000000000000003582876f9735be832000213deA8C6d0dF76Eea870000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
    "alloc": {
        "0x3582876f9735be832000213deA8C6d0dF76Eea87": {
            "balance": "50000000000000000000000000000000000"
        }
    }
}

Expected behaviour

Seal blocks every 2 seconds

Actual behaviour

Doesn't seal any block

Steps to reproduce the behaviour

Running geth locally from genesis file

Backtrace

INFO [03-13|17:38:58.583] Maximum peer count                       ETH=50 total=50
INFO [03-13|17:38:58.584] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [03-13|17:38:58.587] Set global gas cap                       cap=50,000,000
INFO [03-13|17:38:58.587] Initializing the KZG library             backend=gokzg
INFO [03-13|17:38:58.607] Defaulting to pebble as the backing database
INFO [03-13|17:38:58.607] Allocated cache and file handles         database=/.data/geth/chaindata cache=16.00MiB handles=16
INFO [03-13|17:38:58.623] Opened ancient database                  database=/.data/geth/chaindata/ancient/chain readonly=false
INFO [03-13|17:38:58.624] State scheme set by user                 scheme=path
ERROR[03-13|17:38:58.624] Head block is not reachable
ERROR[03-13|17:38:58.624] Zero trie root hash!
INFO [03-13|17:38:58.632] Opened ancient database                  database=/.data/geth/chaindata/ancient/state readonly=false
INFO [03-13|17:38:58.632] Writing custom genesis block
INFO [03-13|17:38:58.658] Successfully wrote genesis state         database=chaindata                           hash=13368e..8b3b54
INFO [03-13|17:38:58.658] Defaulting to pebble as the backing database
INFO [03-13|17:38:58.658] Allocated cache and file handles         database=/.data/geth/lightchaindata cache=16.00MiB handles=16
INFO [03-13|17:38:58.677] Opened ancient database                  database=/.data/geth/lightchaindata/ancient/chain readonly=false
INFO [03-13|17:38:58.677] State scheme set by user                 scheme=path
ERROR[03-13|17:38:58.677] Zero trie root hash!
ERROR[03-13|17:38:58.677] Head block is not reachable
INFO [03-13|17:38:58.684] Opened ancient database                  database=/.data/geth/lightchaindata/ancient/state readonly=false
INFO [03-13|17:38:58.684] Writing custom genesis block
INFO [03-13|17:38:58.705] Successfully wrote genesis state         database=lightchaindata                           hash=13368e..8b3b54
INFO [03-13|17:38:58.808] Starting Geth on Ethereum mainnet...
INFO [03-13|17:38:58.808] Bumping default cache on mainnet         provided=1024 updated=4096
INFO [03-13|17:38:58.810] Maximum peer count                       ETH=50 total=50
INFO [03-13|17:38:58.811] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
WARN [03-13|17:38:58.813] Option --miner.etherbase is deprecated as the etherbase is set by the consensus client post-merge
WARN [03-13|17:38:58.813] The flag --mine is deprecated and will be removed
INFO [03-13|17:38:58.814] Set global gas cap                       cap=50,000,000
INFO [03-13|17:38:58.814] Initializing the KZG library             backend=gokzg
INFO [03-13|17:38:58.837] Allocated trie memory caches             clean=614.00MiB dirty=1024.00MiB
INFO [03-13|17:38:58.838] Using pebble as the backing database
INFO [03-13|17:38:58.838] Allocated cache and file handles         database=/.data/geth/chaindata cache=2.00GiB handles=524,288
INFO [03-13|17:38:58.857] Opened ancient database                  database=/.data/geth/chaindata/ancient/chain readonly=false
INFO [03-13|17:38:58.858] State scheme set by user                 scheme=path
INITIING CLIQUE
INFO [03-13|17:38:58.858] Initialising Ethereum protocol           network=1 dbversion=<nil>
WARN [03-13|17:38:58.859] Sanitizing invalid node buffer size      provided=1024.00MiB updated=256.00MiB
INFO [03-13|17:38:58.859] Failed to load journal, discard it       err="journal not found"
INFO [03-13|17:38:58.862] Opened ancient database                  database=/.data/geth/chaindata/ancient/state readonly=false
INFO [03-13|17:38:58.863]
INFO [03-13|17:38:58.863] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-13|17:38:58.863] Chain ID:  1 (mainnet)
INFO [03-13|17:38:58.863] Consensus: Clique (proof-of-authority)
INFO [03-13|17:38:58.863]
INFO [03-13|17:38:58.863] Pre-Merge hard forks (block based):
INFO [03-13|17:38:58.863]  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [03-13|17:38:58.863]  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [03-13|17:38:58.863]  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [03-13|17:38:58.863]  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [03-13|17:38:58.863]  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [03-13|17:38:58.863]  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [03-13|17:38:58.863]  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [03-13|17:38:58.863]  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [03-13|17:38:58.863]  - Muir Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [03-13|17:38:58.863]  - Berlin:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [03-13|17:38:58.863]  - London:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [03-13|17:38:58.863]  - Arrow Glacier:               #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/arrow-glacier.md)
INFO [03-13|17:38:58.863]  - Gray Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/gray-glacier.md)
INFO [03-13|17:38:58.863]
INFO [03-13|17:38:58.863] The Merge is not yet available for this network!
INFO [03-13|17:38:58.863]  - Hard-fork specification: https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [03-13|17:38:58.863]
INFO [03-13|17:38:58.863] Post-Merge hard forks (timestamp based):
INFO [03-13|17:38:58.863]  - Shanghai:                    @0          (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [03-13|17:38:58.863]
INFO [03-13|17:38:58.863] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [03-13|17:38:58.863]
INFO [03-13|17:38:58.863] Loaded most recent local block           number=0 hash=13368e..8b3b54 td=1 age=2w1d9h
WARN [03-13|17:38:58.863] Failed to load snapshot                  err="missing or corrupted snapshot"
INFO [03-13|17:38:58.866] Rebuilding state snapshot
INFO [03-13|17:38:58.868] Initialized transaction indexer          range="last 2350000 blocks"
INFO [03-13|17:38:58.868] Resuming state snapshot generation       root=41bdae..4a5cb4 accounts=0 slots=0 storage=0.00B dangling=0 elapsed=1.446ms
INFO [03-13|17:38:58.870] Generated state snapshot                 accounts=36 slots=93 storage=9.81KiB dangling=0 elapsed=3.727ms
INFO [03-13|17:38:58.909] Enabled snap sync                        head=0 hash=13368e..8b3b54
INFO [03-13|17:38:58.910] Gasprice oracle is ignoring threshold set threshold=2
WARN [03-13|17:38:58.911] Engine API enabled                       protocol=eth
WARN [03-13|17:38:58.911] Engine API started but chain not configured for merge yet
INFO [03-13|17:38:58.911] Starting peer-to-peer node               instance=Geth/v1.14.0-unstable/linux-arm64/go1.20.10
INFO [03-13|17:38:58.917] IPC endpoint opened                      url=/.data/geth.ipc
INFO [03-13|17:38:58.917] New local node record                    seq=1,710,351,538,917 id=99a95db836932e0e ip=127.0.0.1 udp=0 tcp=30303
INFO [03-13|17:38:58.918] Started P2P networking                   self="enode://56a061fc86fcd6836d7a17fc8358d112906af15c8b39b35ee18aa2f6a7b4a9e280cd6463f0f4b364449db2a4586bf5653dfeb11fcbe3b70c213e71f9ef799f18@127.0.0.1:30303?discport=0"
INFO [03-13|17:38:58.918] Generated JWT secret                     path=/.data/geth/jwtsecret
INFO [03-13|17:38:58.918] HTTP server started                      endpoint=[::]:8545 auth=false prefix= cors= vhosts=localhost
INFO [03-13|17:38:58.918] WebSocket enabled                        url=ws://[::]:8546
INFO [03-13|17:38:58.919] WebSocket enabled                        url=ws://127.0.0.1:8551
INFO [03-13|17:38:58.919] HTTP server started                      endpoint=127.0.0.1:8551 auth=true  prefix= cors=localhost vhosts=localhost
INFO [03-13|17:38:59.375] Unlocked account                         address=0x3582876f9735be832000213deA8C6d0dF76Eea87
INFO [03-13|17:39:00.525] Setting new local account                address=0xf85f1A6051c3a6814868D2d7135d4a3eAd50009b
INFO [03-13|17:39:00.526] Submitted contract creation              hash=0xd8fe7aa21efc94a00ec878426db6f2396f7dae21e081f2aaa5f0f1a5f95df7eb from=0xf85f1A6051c3a6814868D2d7135d4a3eAd50009b nonce=0 contract=0x4524A03bE16a99A1650DC7ae3B48130b458D16c5 value=0
WARN [03-13|17:39:00.530] Served eth_getTransactionReceipt         conn=192.168.65.1:43675 reqid=2 duration="253.5µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [03-13|17:39:00.641] Served eth_getTransactionReceipt         conn=192.168.65.1:43675 reqid=3 duration="278.5µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"
WARN [03-13|17:39:00.754] Served eth_getTransactionReceipt         conn=192.168.65.1:43675 reqid=4 duration="261.292µs" err="transaction indexing is in progress" errdata="transaction indexing is in progress"

I see endless transaction indexing is in progress because the blocks are not sealed.

[fjl]

So the issue here is, the chain config contains "terminalTotalDifficultyPassed": true which indicates that the network is post-merge. With this configuration autonomous block production is disabled in Geth.

However, you should also know that Geth v1.14 will no longer support Clique network operation. It will only be supported in a limited way to allow importing old testnet blocks. If you have a Clique-based network you are supposed to transition it to proof-of-stake using Geth 1.13.x.

[0x366]

Thank you for explanation! Could you please elaborate on how to properly handle this scenario in 1.14:

I was using clique for e2e test (for every test I was spinning an empty blockchain with some preallocated contracts and basically was submitting txs and checking their receipts). It was fitting my purpose 100% as I could just auto seal new block every 1-2 seconds and dont need to run consensus client.

How to archive that in 1.14 ? Besides dropping clique and terminalTotalDifficultyPassed fields, I guess I also need to run consensus client ?

[fjl]

You can use geth --dev for testing. It can be configured with a custom genesis block if needed. See https://geth.ethereum.org/docs/developers/dapp-developer/dev-mode for more info.

[anhnh131]

@fjl Hi, I had a private network with POA consensus (use Clique). I want to upgrade Shanghai for my network, but after investigating, it doesn't support it, right? #26484 (comment)
Currently, this is my Genesis config.

{
  "chainId": 1337,
  "homesteadBlock": 0,
  "eip150Block": 0,
  "eip155Block": 0,
  "eip158Block": 0,
  "byzantiumBlock": 0,
  "constantinopleBlock": 0,
  "petersburgBlock": 0,
  "istanbulBlock": 0,
  "muirGlacierBlock": 0,
  "berlinBlock": 0,
  "londonBlock": 0,
  "clique": { "period": 5, "epoch": 30000 }
}

How do I upgrade Shanghai for my network?