Generate both ERA and ERA1 files for Sepolia testnet #189

jakubgs · 2024-07-11T06:35:00Z

There are two types of chain state archive files:

ERA - Chain history after the merge.
ERA1 - Chain history before the merge.

Currently we are hosting ERA files under https://sepolia.era.nimbus.team/ but some of them are missing:

[email protected]:/data/era % j --since '3 days ago' -u nimbus-era-files-update --grep 'Skipping era' | head
Jul 08 04:29:32 ncli_db[335659]: NTC 2024-07-08 04:29:32.165+00:00 Skipping era, state history not available  tid=335659 file=ncli_db.nim:616 era=0 name=sepolia-00000-d8ea171f.era
Jul 08 04:29:32 ncli_db[335659]: NTC 2024-07-08 04:29:32.165+00:00 Skipping era, state history not available  tid=335659 file=ncli_db.nim:602 era=1 tail=d4d3efba:4332543
Jul 08 04:29:32 ncli_db[335659]: NTC 2024-07-08 04:29:32.165+00:00 Skipping era, state history not available  tid=335659 file=ncli_db.nim:602 era=2 tail=d4d3efba:4332543
Jul 08 04:29:32 ncli_db[335659]: NTC 2024-07-08 04:29:32.165+00:00 Skipping era, state history not available  tid=335659 file=ncli_db.nim:602 era=3 tail=d4d3efba:4332543
Jul 08 04:29:32 ncli_db[335659]: NTC 2024-07-08 04:29:32.165+00:00 Skipping era, state history not available  tid=335659 file=ncli_db.nim:602 era=4 tail=d4d3efba:4332543

Unfortunately our Beacon Nodes are running with history mode set to prune, which means they only retain last 6 months of data. In order to re-generate this new data we need to purge BN data, and switch the history mode to archive, and let it re-sync. The current available storage does not make that possible since a fully synced BN might take up up to 1 TB:

[email protected]:~ % df -h /docker /data
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc        2.9T  2.3T  491G  83% /docker
/dev/sdb        1.5T  767G  625G  56% /data

For this reason we need to perform these steps to get what we want hopefully without purchasing extra storage:

Generate ERA1 files from a single Geth node.

geth --datadir=/docker/geth-sepolia-stable/node/data --sepolia export-history /data/era1 0 1450409

Switch Geth nodes to use snap mode with --syncmode to reclaim space.
- Just changing the flag might not reclaim the space, a purge of container data might be necessary.
Move /data/era files to /docker to recover space necessary for BN re-sync.
- The Nginx configuration needs to be modified to continue the public hosting of ERA files.
Re-sync a single stable BN with history retention set to archive using reclaimed space.
- This is controlled by the history configuration flag which can be prune or archive.
Re-generate missing ERA files from that BN using nimbus-era-files-update.service.
Switch all BNs back to prune history mode to reclaim space.

Once done the ERA1 files should also be hosted. Probably under: https://sepolia.era1.nimbus.team/

This is a low priority task.

The text was updated successfully, but these errors were encountered:

jakubgs · 2024-07-18T11:36:06Z

If Geth re-sync is necessary, it should be done one node at a time, preferably with the respective BN using a different Geth node.

jakubgs · 2024-07-18T11:42:11Z

The simplest way to check Geth and Beacon Node sync state is using the RPC and REST APIs respectively:

[email protected]:~ % for dir in /docker/geth-*; do $dir/rpc.sh eth_syncing | jq -c; done
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}
{"jsonrpc":"2.0","id":1,"result":false}

[email protected]:~ % for port in $(seq 9311 9314); do c 0:$port/eth/v1/node/syncing | jq -c; done
{"data":{"head_slot":"5464109","sync_distance":"0","is_syncing":false,"is_optimistic":false,"el_offline":false}}
{"data":{"head_slot":"5464109","sync_distance":"0","is_syncing":false,"is_optimistic":false,"el_offline":false}}
{"data":{"head_slot":"5464109","sync_distance":"0","is_syncing":false,"is_optimistic":false,"el_offline":false}}
{"data":{"head_slot":"5464109","sync_distance":"0","is_syncing":false,"is_optimistic":false,"el_offline":false}}

jakubgs · 2024-07-18T11:45:34Z

The second way to check Geth and Beacon Node sync stage is using metrics in Grafana:
https://grafana.infra.status.im/d/VyUgFRWZz/geth-syncing

Unfortunately on the Sepolia host the Geth containers do not have an accompanying exporter container:

[email protected]:~ % docker ps
CONTAINER ID   NAMES                        IMAGE                        CREATED       STATUS
f884dbd7292a   geth-sepolia-libp2p-node     ethereum/client-go:v1.14.6   12 days ago   Up 12 days
c8a8b3672586   geth-sepolia-unstable-node   ethereum/client-go:v1.14.6   12 days ago   Up 12 days
b2cf759f6868   geth-sepolia-testing-node    ethereum/client-go:v1.14.6   12 days ago   Up 12 days
71b7d6c720ff   geth-sepolia-stable-node     ethereum/client-go:v1.14.6   12 days ago   Up 12 days

You don't have to deploy them, but they can be useful when long-term re-sync processes run. You can see how it's set up here:

infra-nimbus/ansible/group_vars/nimbus.holesky.yml

Lines 31 to 35 in 4555e37

    
           # Geth metrics¬ 
        
           geth_expo_service_name: '{{ geth_service_name }}' 
        
           geth_expo_source_cont_name: '{{ geth_cont_name }}' 
        
           geth_expo_source_data_path: '{{ geth_cont_vol }}/data' 
        
           geth_expo_cont_port:  '{{ 9400 + (idx|int) + 1 }}'

infra-nimbus/ansible/holesky.yml

Lines 58 to 65 in 4555e37

    
           - include_role: 
        
               name: infra-role-geth-exporter 
        
               apply: 
        
                 tags: always 
        
             tags: [ geth-exporter ] 
        
             when: node.el == "geth" 
        
             with_items: '{{ nodes_layout[hostname] }}' 
        
             loop_control: { loop_var: node, index_var: idx }

jakubgs · 2024-07-18T11:48:37Z

The way to view Beacon Node sync progress is also to use Grafana:

https://grafana.infra.status.im/d/pgeNfj2Wz23/nimbus-beacon-node?orgId=1&var-instance=linux-01.ih-eu-mda1.nimbus.sepolia&var-container=beacon-node-sepolia-stable

These metrics require no additional setup.

jakubgs · 2024-07-18T11:50:26Z

These metrics are of course available on the host itself by using the correct metrics service port:

[email protected]:~ % grep metrics /data/beacon-node-sepolia-stable/conf/config.toml
### Metrics
metrics = true
metrics-address = "0.0.0.0"
metrics-port = 9211

[email protected]:~ % curl -s 0:9211/metrics | grep -e beacon_slot -e beacon_head_slot
# HELP beacon_head_slot Slot of the head block of the beacon chain
# TYPE beacon_head_slot gauge
beacon_head_slot 5464150.0
beacon_head_slot_created 1719273604.0
# HELP beacon_slot Latest slot of the beacon chain state
# TYPE beacon_slot gauge
beacon_slot 5464150.0
beacon_slot_created 1719273604.0

jakubgs · 2024-07-18T11:54:39Z

If you want to check Geth docker container configs you need to look at their Docker Compose files:

[email protected]:~ % grep http.port /docker/geth-sepolia-stable/docker-compose.yml 
      --http.port=8556

And if you want to check Beacon Node systemd service configs you need to look at the service and the config file:

[email protected]:~ % sudo systemctl cat beacon-node-sepolia-stable | grep config-file  
    --config-file=/data/beacon-node-sepolia-stable/conf/config.toml

[email protected]:~ % grep history /data/beacon-node-sepolia-stable/conf/config.toml
history = "prune"

jakubgs · 2024-07-18T11:57:43Z

The main configuration files for any Nimbus fleets are in a few locations, here's a list for Sepolia:

holesky.yml - Playbook
group_vars/nimbus.sepolia.yml - Group Variables
vars/layout/sepolia.yml - Node Layout
host_vars/linux-01.ih-eu-mda1.nimbus.sepolia.yml - Host Variables

siddarthkay · 2024-07-26T08:25:44Z

Logging size after changing sync mode of libp2p sepolia to snap sync manually ( not via ansible ).

sudo du -hsc /docker/*
591G	/docker/geth-sepolia-libp2p
599G	/docker/geth-sepolia-stable
591G	/docker/geth-sepolia-testing
591G	/docker/geth-sepolia-unstable
6.5M	/docker/log
16K	/docker/lost+found
2.4T	total

Will check in a bit if size is purged or not.

siddarthkay · 2024-07-26T08:27:50Z

I ( with the help of Jakub) generated the ERA1 files like this :

sudo geth --datadir=/docker/geth-sepolia-stable/node/data --sepolia export-history /data/era1 0 1450409
INFO [07-26|08:07:06.297] Maximum peer count                       ETH=50 total=50
INFO [07-26|08:07:06.299] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [07-26|08:07:06.303] Set global gas cap                       cap=50,000,000
INFO [07-26|08:07:06.303] Initializing the KZG library             backend=gokzg
INFO [07-26|08:07:06.479] Using leveldb as the backing database
INFO [07-26|08:07:06.479] Allocated cache and file handles         database=/docker/geth-sepolia-stable/node/data/geth/chaindata cache=512.00MiB handles=524,288 readonly=true
INFO [07-26|08:07:07.506] Using LevelDB as the backing database
INFO [07-26|08:07:07.565] Opened ancient database                  database=/docker/geth-sepolia-stable/node/data/geth/chaindata/ancient/chain readonly=true
INFO [07-26|08:07:07.574] State scheme set to already existing     scheme=hash
INFO [07-26|08:07:07.575]
INFO [07-26|08:07:07.575] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [07-26|08:07:07.575] Chain ID:  11155111 (sepolia)
INFO [07-26|08:07:07.575] Consensus: Beacon (proof-of-stake), merged from Ethash (proof-of-work)
INFO [07-26|08:07:07.575]
INFO [07-26|08:07:07.575] Pre-Merge hard forks (block based):
INFO [07-26|08:07:07.575]  - Homestead:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/homestead.md)
INFO [07-26|08:07:07.575]  - Tangerine Whistle (EIP 150): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/tangerine-whistle.md)
INFO [07-26|08:07:07.575]  - Spurious Dragon/1 (EIP 155): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [07-26|08:07:07.575]  - Spurious Dragon/2 (EIP 158): #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/spurious-dragon.md)
INFO [07-26|08:07:07.575]  - Byzantium:                   #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/byzantium.md)
INFO [07-26|08:07:07.575]  - Constantinople:              #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/constantinople.md)
INFO [07-26|08:07:07.575]  - Petersburg:                  #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/petersburg.md)
INFO [07-26|08:07:07.575]  - Istanbul:                    #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/istanbul.md)
INFO [07-26|08:07:07.575]  - Muir Glacier:                #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/muir-glacier.md)
INFO [07-26|08:07:07.575]  - Berlin:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/berlin.md)
INFO [07-26|08:07:07.575]  - London:                      #0        (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/london.md)
INFO [07-26|08:07:07.575]
INFO [07-26|08:07:07.575] Merge configured:
INFO [07-26|08:07:07.575]  - Hard-fork specification:    https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/paris.md
INFO [07-26|08:07:07.575]  - Network known to be merged: true
INFO [07-26|08:07:07.575]  - Total terminal difficulty:  17000000000000000
INFO [07-26|08:07:07.575]  - Merge netsplit block:       #1735371
INFO [07-26|08:07:07.575]
INFO [07-26|08:07:07.575] Post-Merge hard forks (timestamp based):
INFO [07-26|08:07:07.575]  - Shanghai:                    @1677557088 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/shanghai.md)
INFO [07-26|08:07:07.575]  - Cancun:                      @1706655072 (https://github.com/ethereum/execution-specs/blob/master/network-upgrades/mainnet-upgrades/cancun.md)
INFO [07-26|08:07:07.575]
INFO [07-26|08:07:07.575] ---------------------------------------------------------------------------------------------------------------------------------------------------------
INFO [07-26|08:07:07.575]
INFO [07-26|08:07:07.579] Loaded most recent local block           number=6,377,982 hash=3337b9..40c0ba td=17,000,018,015,853,232 age=20m31s
INFO [07-26|08:07:07.580] Loaded most recent local finalized block number=6,377,916 hash=ede23c..11b0da td=17,000,018,015,853,232 age=37m31s
INFO [07-26|08:07:07.664] Exporting blockchain history             dir=/data/era1
INFO [07-26|08:07:17.123] Exporting blocks                         exported=24576 elapsed=9.458s
INFO [07-26|08:07:25.334] Exporting blocks                         exported=57344 elapsed=17.670s
INFO [07-26|08:07:33.841] Exporting blocks                         exported=98304 elapsed=26.177s
INFO [07-26|08:07:41.959] Exporting blocks                         exported=147,456 elapsed=34.294s
INFO [07-26|08:07:51.023] Exporting blocks                         exported=212,992 elapsed=43.358s
INFO [07-26|08:07:59.636] Exporting blocks                         exported=278,528 elapsed=51.972s
INFO [07-26|08:08:08.076] Exporting blocks                         exported=344,064 elapsed=1m0.412s
INFO [07-26|08:08:16.402] Exporting blocks                         exported=409,600 elapsed=1m8.737s
INFO [07-26|08:08:24.805] Exporting blocks                         exported=475,136 elapsed=1m17.140s
INFO [07-26|08:08:33.030] Exporting blocks                         exported=540,672 elapsed=1m25.366s
INFO [07-26|08:08:41.351] Exporting blocks                         exported=606,208 elapsed=1m33.686s
INFO [07-26|08:08:49.398] Exporting blocks                         exported=671,744 elapsed=1m41.734s
INFO [07-26|08:08:57.796] Exporting blocks                         exported=737,280 elapsed=1m50.131s
INFO [07-26|08:09:06.173] Exporting blocks                         exported=802,816 elapsed=1m58.509s
INFO [07-26|08:09:14.859] Exporting blocks                         exported=868,352 elapsed=2m7.194s
INFO [07-26|08:09:22.993] Exporting blocks                         exported=933,888 elapsed=2m15.329s
INFO [07-26|08:09:31.839] Exporting blocks                         exported=1,007,616 elapsed=2m24.175s
INFO [07-26|08:09:39.842] Exporting blocks                         exported=1,073,152 elapsed=2m32.177s
INFO [07-26|08:09:48.293] Exporting blocks                         exported=1,138,688 elapsed=2m40.629s
INFO [07-26|08:09:57.139] Exporting blocks                         exported=1,204,224 elapsed=2m49.475s
INFO [07-26|08:10:06.043] Exporting blocks                         exported=1,261,568 elapsed=2m58.378s
INFO [07-26|08:10:14.383] Exporting blocks                         exported=1,310,720 elapsed=3m6.718s
INFO [07-26|08:10:22.433] Exporting blocks                         exported=1,368,064 elapsed=3m14.769s
INFO [07-26|08:10:32.205] Exporting blocks                         exported=1,417,216 elapsed=3m24.541s
INFO [07-26|08:10:36.807] Exported blockchain to                   dir=/data/era1
Export done in 3m29.143025774s

siddarthkay · 2024-07-26T08:30:00Z

@advaita-saha : could you please take a look at these generated ERA 1 files and let us know if we missed something?

sudo du -hsc era1/*
12K	era1/checksums.txt
3.2M	era1/sepolia-00000-643a00f7.era1
3.2M	era1/sepolia-00001-f3ea493e.era1
3.2M	era1/sepolia-00002-dcb01f4d.era1
3.2M	era1/sepolia-00003-18b32584.era1
3.2M	era1/sepolia-00004-e93b3f11.era1
3.2M	era1/sepolia-00005-90918472.era1
3.2M	era1/sepolia-00006-a4a583b1.era1
3.2M	era1/sepolia-00007-ef1e0a86.era1
3.2M	era1/sepolia-00008-dfe1e6dd.era1
3.2M	era1/sepolia-00009-dba60c04.era1
3.2M	era1/sepolia-00010-c99c706d.era1
3.2M	era1/sepolia-00011-04513853.era1
3.2M	era1/sepolia-00012-6e07302d.era1
3.2M	era1/sepolia-00013-25709be2.era1
3.2M	era1/sepolia-00014-7c90e5a5.era1
3.2M	era1/sepolia-00015-fadc08ab.era1
3.2M	era1/sepolia-00016-d44b2499.era1
3.2M	era1/sepolia-00017-02451eb3.era1
3.2M	era1/sepolia-00018-cdb53b11.era1
3.2M	era1/sepolia-00019-fa770019.era1
3.2M	era1/sepolia-00020-15fc80e7.era1
3.6M	era1/sepolia-00021-b8814b14.era1
5.0M	era1/sepolia-00022-87944c43.era1
3.3M	era1/sepolia-00023-516a30f5.era1
3.2M	era1/sepolia-00024-b922dde6.era1
3.3M	era1/sepolia-00025-a081e175.era1
3.3M	era1/sepolia-00026-d3931a12.era1
3.4M	era1/sepolia-00027-43f5e0ac.era1
3.5M	era1/sepolia-00028-dae08170.era1
3.4M	era1/sepolia-00029-17818143.era1
3.4M	era1/sepolia-00030-cc7fc4eb.era1
3.4M	era1/sepolia-00031-3171eede.era1
3.4M	era1/sepolia-00032-9e2ff5ca.era1
3.7M	era1/sepolia-00033-bc921023.era1
3.6M	era1/sepolia-00034-b2496634.era1
3.4M	era1/sepolia-00035-f3add4f1.era1
3.4M	era1/sepolia-00036-0db12924.era1
3.4M	era1/sepolia-00037-ae8bdd13.era1
3.7M	era1/sepolia-00038-81f80b03.era1
3.5M	era1/sepolia-00039-03e5d6f1.era1
3.5M	era1/sepolia-00040-296fe287.era1
3.5M	era1/sepolia-00041-5cd87470.era1
3.5M	era1/sepolia-00042-de86936c.era1
3.5M	era1/sepolia-00043-ede4e682.era1
3.5M	era1/sepolia-00044-4d49a81a.era1
3.5M	era1/sepolia-00045-7071c19b.era1
3.5M	era1/sepolia-00046-4f4fe79f.era1
3.5M	era1/sepolia-00047-76b58fb3.era1
3.4M	era1/sepolia-00048-0fa9d93c.era1
3.4M	era1/sepolia-00049-d193ff47.era1
3.4M	era1/sepolia-00050-736b969f.era1
3.4M	era1/sepolia-00051-466eb482.era1
3.4M	era1/sepolia-00052-9752212c.era1
3.4M	era1/sepolia-00053-b2897233.era1
3.3M	era1/sepolia-00054-aedad6a8.era1
3.3M	era1/sepolia-00055-19af5091.era1
3.3M	era1/sepolia-00056-9ac921a1.era1
3.4M	era1/sepolia-00057-240f011a.era1
3.4M	era1/sepolia-00058-06d606d6.era1
3.3M	era1/sepolia-00059-8ea69a55.era1
3.3M	era1/sepolia-00060-d22079b0.era1
3.3M	era1/sepolia-00061-834d00e3.era1
3.4M	era1/sepolia-00062-d1230cc2.era1
3.3M	era1/sepolia-00063-e1e9fa40.era1
3.3M	era1/sepolia-00064-7f827781.era1
3.4M	era1/sepolia-00065-9d8993d6.era1
3.4M	era1/sepolia-00066-8d516260.era1
3.4M	era1/sepolia-00067-7466141b.era1
3.4M	era1/sepolia-00068-1bd11e8a.era1
3.4M	era1/sepolia-00069-536ecd2c.era1
3.4M	era1/sepolia-00070-b27f7c5c.era1
3.4M	era1/sepolia-00071-d2ef5349.era1
3.4M	era1/sepolia-00072-b23578a9.era1
3.4M	era1/sepolia-00073-8bacb416.era1
3.4M	era1/sepolia-00074-0e81003c.era1
3.4M	era1/sepolia-00075-09177034.era1
3.4M	era1/sepolia-00076-0f2898a3.era1
3.4M	era1/sepolia-00077-61de3538.era1
3.4M	era1/sepolia-00078-be407b86.era1
3.4M	era1/sepolia-00079-d5c957be.era1
3.4M	era1/sepolia-00080-ae6ce310.era1
3.4M	era1/sepolia-00081-daac26d6.era1
3.4M	era1/sepolia-00082-ecb8f0c2.era1
3.4M	era1/sepolia-00083-5b017cd4.era1
3.4M	era1/sepolia-00084-8c1f92cd.era1
3.4M	era1/sepolia-00085-9180667c.era1
3.4M	era1/sepolia-00086-1aca7add.era1
3.5M	era1/sepolia-00087-0e100944.era1
3.4M	era1/sepolia-00088-fcdd7aa5.era1
3.4M	era1/sepolia-00089-b23b2368.era1
3.5M	era1/sepolia-00090-b78edeb7.era1
3.4M	era1/sepolia-00091-4c81cdca.era1
3.5M	era1/sepolia-00092-ddbf3a4c.era1
3.7M	era1/sepolia-00093-a52a45ec.era1
3.5M	era1/sepolia-00094-839ea4e5.era1
3.5M	era1/sepolia-00095-bccc0958.era1
3.6M	era1/sepolia-00096-429b2f39.era1
3.5M	era1/sepolia-00097-6f74184d.era1
3.5M	era1/sepolia-00098-40286059.era1
3.9M	era1/sepolia-00099-dd1ef4f4.era1
6.4M	era1/sepolia-00100-bd3d6bf4.era1
5.6M	era1/sepolia-00101-41676e0d.era1
3.5M	era1/sepolia-00102-4f3c1fe1.era1
3.5M	era1/sepolia-00103-e13a95f0.era1
3.4M	era1/sepolia-00104-4e49b068.era1
3.4M	era1/sepolia-00105-4ab7d9ea.era1
3.4M	era1/sepolia-00106-5ec678af.era1
3.5M	era1/sepolia-00107-44721c55.era1
3.4M	era1/sepolia-00108-3b4ad768.era1
3.5M	era1/sepolia-00109-16e54758.era1
3.5M	era1/sepolia-00110-b5da103b.era1
3.5M	era1/sepolia-00111-15c53aba.era1
3.5M	era1/sepolia-00112-8e2144e3.era1
3.5M	era1/sepolia-00113-1e1aac0a.era1
3.4M	era1/sepolia-00114-0d7a4b23.era1
3.4M	era1/sepolia-00115-8e983a59.era1
3.4M	era1/sepolia-00116-1708ba6e.era1
3.4M	era1/sepolia-00117-86962046.era1
3.4M	era1/sepolia-00118-934e9f5f.era1
3.4M	era1/sepolia-00119-4d88db3e.era1
3.5M	era1/sepolia-00120-1a3274d5.era1
3.5M	era1/sepolia-00121-62b79757.era1
3.4M	era1/sepolia-00122-41d14652.era1
3.4M	era1/sepolia-00123-67f86708.era1
3.4M	era1/sepolia-00124-488cb226.era1
3.5M	era1/sepolia-00125-8a21f7da.era1
3.5M	era1/sepolia-00126-6569478a.era1
3.4M	era1/sepolia-00127-1312ab23.era1
3.4M	era1/sepolia-00128-1c95bdc6.era1
3.4M	era1/sepolia-00129-f32fdefa.era1
3.4M	era1/sepolia-00130-8f31c590.era1
3.4M	era1/sepolia-00131-5aaad354.era1
3.4M	era1/sepolia-00132-2147d970.era1
3.4M	era1/sepolia-00133-690cc16f.era1
3.4M	era1/sepolia-00134-fe276e6b.era1
3.5M	era1/sepolia-00135-ce40efd0.era1
3.6M	era1/sepolia-00136-adc6e1bb.era1
3.8M	era1/sepolia-00137-805a333a.era1
3.8M	era1/sepolia-00138-65ff170f.era1
3.8M	era1/sepolia-00139-1c86d86a.era1
3.7M	era1/sepolia-00140-e7f36280.era1
5.8M	era1/sepolia-00141-5d3d0d5a.era1
3.9M	era1/sepolia-00142-e45aa418.era1
4.0M	era1/sepolia-00143-70379f59.era1
4.8M	era1/sepolia-00144-c345e75c.era1
5.4M	era1/sepolia-00145-9f4bb4c5.era1
4.2M	era1/sepolia-00146-3336d26b.era1
3.9M	era1/sepolia-00147-6b007873.era1
4.2M	era1/sepolia-00148-35f8a5b4.era1
4.2M	era1/sepolia-00149-01b6e9ca.era1
4.2M	era1/sepolia-00150-0c372e80.era1
4.0M	era1/sepolia-00151-c15a5955.era1
5.2M	era1/sepolia-00152-ef7ac893.era1
10M	era1/sepolia-00153-0e4073c2.era1
11M	era1/sepolia-00154-298c3549.era1
7.4M	era1/sepolia-00155-a99309bb.era1
9.4M	era1/sepolia-00156-8c33148f.era1
7.5M	era1/sepolia-00157-ab02fc10.era1
6.1M	era1/sepolia-00158-17992856.era1
7.1M	era1/sepolia-00159-b3448cf6.era1
5.8M	era1/sepolia-00160-acad7054.era1
6.4M	era1/sepolia-00161-36e611bd.era1
5.4M	era1/sepolia-00162-3557cc58.era1
3.6M	era1/sepolia-00163-638b9afe.era1
3.7M	era1/sepolia-00164-fbc67b64.era1
4.0M	era1/sepolia-00165-442b83a1.era1
5.5M	era1/sepolia-00166-018915fc.era1
5.6M	era1/sepolia-00167-0df2b4c5.era1
4.5M	era1/sepolia-00168-7cab66f5.era1
9.2M	era1/sepolia-00169-75850131.era1
13M	era1/sepolia-00170-c2044b78.era1
14M	era1/sepolia-00171-af44ff2e.era1
15M	era1/sepolia-00172-bd76a8d9.era1
16M	era1/sepolia-00173-b6924da5.era1
11M	era1/sepolia-00174-2a61fac5.era1
14M	era1/sepolia-00175-1cc76405.era1
9.1M	era1/sepolia-00176-20daa2c6.era1
312K	era1/sepolia-00177-d0c55383.era1
734M	total

jakubgs · 2024-07-26T08:37:22Z

Since it seems like switching syncmode to snap on a Geth node does not prune old data we might need to do a resync.

To do a Geth resync we need to:

Change BN Systemd service config so it uses an additional EL node that is not re-syncing.
- You can check Grafana dashboard to see if it's making requests to both.
Stop the relevant Geth container.
Clean up contents of /docker/geth-sepolia-libp2p/node/data/geth(except for the nodekey file).
Start the relevant Geth container.
Wait until it re-sync. This can be checked with /docker/geth-sepolia-libp2p/rpc.sh eth_syncing.
- Verify on Beacon Chain dashboard that the validators are online.
Change BN Systemd service config back to using just their dedicated Geth node.

This way we can avoid unnecessary downtime for the validators.

advaita-saha · 2024-07-26T10:15:41Z

@siddarthkay the Era1 files looks good to me

jakubgs · 2024-07-29T07:40:01Z

@siddarthkay you can add hosting for ERA1 files the same way we do it for ERA files:

infra-nimbus/ansible/host_vars/linux-01.ih-eu-mda1.nimbus.sepolia.yml

Lines 30 to 44 in 2275872

    
           nginx_sites: 
        
             era_files: 
        
               - listen 80 default_server 
        
               - listen 443 ssl 
        
               - server_name {{ era_files_domain }} 
        
               - ssl_certificate     /certs/nimbus.team/origin.crt 
        
               - ssl_certificate_key /certs/nimbus.team/origin.key 
        
               - location / { 
        
                   root {{ era_files_path }}; 
        
                   autoindex on; 
        
                   autoindex_format html; 
        
                 }

Change: 458652e

I probably would just add an extra domain, for example sepolia.era1.nimbus.team.

siddarthkay · 2024-07-29T18:09:09Z

I manually re-synced geth-sepolia-libp2p and got savings of ~ 240 GB

> sudo du -hsc /docker/*
351G	/docker/geth-sepolia-libp2p
604G	/docker/geth-sepolia-stable
596G	/docker/geth-sepolia-testing
596G	/docker/geth-sepolia-unstable
6.5M	/docker/log
16K	/docker/lost+found
2.1T	total

> /docker/geth-sepolia-libp2p/rpc.sh eth_syncing
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": false
}

I'll continue to re-sync other nodes tomorrow

siddarthkay · 2024-07-30T18:11:53Z

I've manually re-synced geth-sepolia-testing

> /docker/geth-sepolia-testing/rpc.sh eth_syncing
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": false
}

> sudo du -hsc /docker/*
352G	/docker/geth-sepolia-libp2p
605G	/docker/geth-sepolia-stable
360G	/docker/geth-sepolia-testing
598G	/docker/geth-sepolia-unstable
6.9M	/docker/log
16K	/docker/lost+found
1.9T	total

siddarthkay · 2024-07-31T05:27:02Z

I've re-synced geth-sepolia-unstable and geth-sepolia-stable

Total space cleaned up is -> 591 + 599 + 591 + 591 - (353 + 361 + 362 + 361) = 935 GB

> /docker/geth-sepolia-stable/rpc.sh eth_syncing
{
  "jsonrpc": "2.0",
  "id": 1,
  "result": false
}

> sudo du -hsc /docker/*
353G	/docker/geth-sepolia-libp2p
361G	/docker/geth-sepolia-stable
362G	/docker/geth-sepolia-testing
361G	/docker/geth-sepolia-unstable
7.7M	/docker/log
16K	/docker/lost+found
1.5T	total

jakubgs · 2024-08-02T07:19:05Z

Now it's time to resync a Beacon Node right?

siddarthkay · 2024-08-03T12:22:17Z

@jakubgs : correct, I'll first move era and era1 files to /docker to save up some space and make sure hosting them still works before I do a BN sync.

> sudo du -hsc /data/*
133G	/data/beacon-node-sepolia-libp2p
133G	/data/beacon-node-sepolia-stable
134G	/data/beacon-node-sepolia-testing
137G	/data/beacon-node-sepolia-unstable
243G	/data/era
734M	/data/era1
16K	/data/lost+found
5.2G	/data/validator-client-sepolia-stable
784G	total

era and era1 files combined take up roughly 244 GB of space

siddarthkay · 2024-08-03T12:59:23Z

I've moved era and era1 files to docker and manually made changes to nginx config on sepolia host for
https://sepolia.era1.nimbus.team/ and for https://sepolia.era.nimbus.team/

> sudo du -hsc /data/*
133G	/data/beacon-node-sepolia-libp2p
133G	/data/beacon-node-sepolia-stable
134G	/data/beacon-node-sepolia-testing
137G	/data/beacon-node-sepolia-unstable
16K	/data/lost+found
5.2G	/data/validator-client-sepolia-stable
540G	total

> df -h
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           6.3G  2.7M  6.3G   1% /run
/dev/sda1       366G   20G  328G   6% /
tmpfs            32G  328K   32G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/sdc        2.9T  1.7T  1.1T  61% /docker
/dev/sdb        1.5T  563G  830G  41% /data
tmpfs           6.3G     0  6.3G   0% /run/user/7009

siddarthkay · 2024-08-06T07:20:20Z

I've re-synced stable beacon node and it took roughly 3 days to sync.

The fully synced node takes ~ 300 GB of space.

> sudo du -hsc /data/*
133G	/data/beacon-node-sepolia-libp2p
295G	/data/beacon-node-sepolia-stable
134G	/data/beacon-node-sepolia-testing
137G	/data/beacon-node-sepolia-unstable
16K	/data/lost+found
5.2G	/data/validator-client-sepolia-stable
703G	total

Now time to generate missing ERA files.

siddarthkay · 2024-08-06T09:44:50Z

I modified /etc/systemd/system/nimbus-era-files-update.service to refer to stable beacon node's db, this was earlier referring to unstable.

I then had to :

> sudo systemctl daemon-reload
> sudo systemctl restart nimbus-era-files-update.service

And then I could see that missing ERA file generation was successful

sudo systemctl status nimbus-era-files-update.service
○ nimbus-era-files-update.service - Update Nimbus beacon node ERA files
     Loaded: loaded (/etc/systemd/system/nimbus-era-files-update.service; static)
     Active: inactive (dead) since Tue 2024-08-06 09:31:41 UTC; 1min 7s ago
TriggeredBy: ● nimbus-era-files-update.timer
       Docs: https://github.com/status-im/infra-role-systemd-timer
    Process: 2590786 ExecStart=/data/beacon-node-sepolia-stable/repo/build/ncli_db 
    --network=sepolia exportEra --db=/data/beacon-node-sepolia-stable/data/db (code=exited, status=0/SUCCESS)
   Main PID: 2590786 (code=exited, status=0/SUCCESS)
        CPU: 1min 23.373s

Aug 06 09:31:41 linux-01.ih-eu-mda1.nimbus.sepolia ncli_db[2590786]: DBG 2024-08-06 09:31:41.499+00:00 
Era file already exists                    tid=2590786 file=ncli_db.nim:552 era=682 name=>

Aug 06 09:31:41 linux-01.ih-eu-mda1.nimbus.sepolia ncli_db[2590786]: DBG 2024-08-06 09:31:41.499+00:00 
Era file already exists                    tid=2590786 file=ncli_db.nim:552 era=683 name=>

Aug 06 09:31:41 linux-01.ih-eu-mda1.nimbus.sepolia ncli_db[2590786]: NTC 2024-08-06 09:31:41.499+00:00 
Written all complete eras                  tid=2590786 file=ncli_db.nim:539 era=684 endSl>

siddarthkay · 2024-08-06T09:50:24Z

I've set history back to prune for sepolia stable beacon node and restarted the service for the config change to take effect.

> cat /data/beacon-node-sepolia-stable/conf/config.toml | grep history
history = "prune"

siddarthkay · 2024-08-10T03:42:57Z

re-syncing completed last night.

and we've reclaimed the space.

> sudo du -hsc /data/*
133G	/data/beacon-node-sepolia-libp2p
118G	/data/beacon-node-sepolia-stable
134G	/data/beacon-node-sepolia-testing
137G	/data/beacon-node-sepolia-unstable
260G	/data/era
734M	/data/era1
16K	/data/lost+found
5.2G	/data/validator-client-sepolia-stable
785G	total

Thanks @jakubgs, this task gave me good insights and made me a bit comfortable with the nimbus fleet.
I consider this task as done.

arnetheduck · 2024-10-03T06:08:01Z

the archive at https://sepolia.era1.nimbus.team/ is incomplete - there should be 183 files cc @siddarthkay

siddarthkay · 2024-10-08T08:44:17Z

@arnetheduck : https://sepolia.era1.nimbus.team/ should be complete now, confirmed with @advaita-saha

I regenerated ERA1 files like this

geth --datadir=/docker/geth-sepolia-stable/node/data --sepolia export-history /data/era1 0 1490944

jakubgs assigned siddarthkay Jul 11, 2024

siddarthkay closed this as completed Aug 10, 2024

yakimant mentioned this issue Aug 30, 2024

Run multiple nimbus-eth1 mainnet instances #193

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate both ERA and ERA1 files for Sepolia testnet #189

Generate both ERA and ERA1 files for Sepolia testnet #189

jakubgs commented Jul 11, 2024 •

edited by siddarthkay

Loading

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024 •

edited

Loading

siddarthkay commented Jul 26, 2024 •

edited

Loading

siddarthkay commented Jul 26, 2024

siddarthkay commented Jul 26, 2024

jakubgs commented Jul 26, 2024 •

edited

Loading

advaita-saha commented Jul 26, 2024

jakubgs commented Jul 29, 2024

siddarthkay commented Jul 29, 2024

siddarthkay commented Jul 30, 2024

siddarthkay commented Jul 31, 2024 •

edited

Loading

jakubgs commented Aug 2, 2024

siddarthkay commented Aug 3, 2024 •

edited

Loading

siddarthkay commented Aug 3, 2024

siddarthkay commented Aug 6, 2024

siddarthkay commented Aug 6, 2024

siddarthkay commented Aug 6, 2024

siddarthkay commented Aug 10, 2024

arnetheduck commented Oct 3, 2024

siddarthkay commented Oct 8, 2024

Generate both ERA and ERA1 files for Sepolia testnet #189

Generate both ERA and ERA1 files for Sepolia testnet #189

Comments

jakubgs commented Jul 11, 2024 • edited by siddarthkay Loading

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024

jakubgs commented Jul 18, 2024 • edited Loading

siddarthkay commented Jul 26, 2024 • edited Loading

siddarthkay commented Jul 26, 2024

siddarthkay commented Jul 26, 2024

jakubgs commented Jul 26, 2024 • edited Loading

advaita-saha commented Jul 26, 2024

jakubgs commented Jul 29, 2024

siddarthkay commented Jul 29, 2024

siddarthkay commented Jul 30, 2024

siddarthkay commented Jul 31, 2024 • edited Loading

jakubgs commented Aug 2, 2024

siddarthkay commented Aug 3, 2024 • edited Loading

siddarthkay commented Aug 3, 2024

siddarthkay commented Aug 6, 2024

siddarthkay commented Aug 6, 2024

siddarthkay commented Aug 6, 2024

siddarthkay commented Aug 10, 2024

arnetheduck commented Oct 3, 2024

siddarthkay commented Oct 8, 2024

jakubgs commented Jul 11, 2024 •

edited by siddarthkay

Loading

jakubgs commented Jul 18, 2024 •

edited

Loading

siddarthkay commented Jul 26, 2024 •

edited

Loading

jakubgs commented Jul 26, 2024 •

edited

Loading

siddarthkay commented Jul 31, 2024 •

edited

Loading

siddarthkay commented Aug 3, 2024 •

edited

Loading