Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A new node can not discover all the active peers that boot node connected #164

Closed
qzhodl opened this issue Dec 21, 2023 · 14 comments
Closed
Assignees
Labels
question Further information is requested
Milestone

Comments

@qzhodl
Copy link
Collaborator

qzhodl commented Dec 21, 2023

On devnet-2, the bootnode was connected with 13 active peers, then we launched a new node, and it can only connect with 7 active peers. We may need to investigate the root cause.

@qzhodl qzhodl added the question Further information is requested label Dec 21, 2023
@qzhodl qzhodl added this to the Devnet-3 milestone Dec 27, 2023
@qzhodl
Copy link
Collaborator Author

qzhodl commented Dec 29, 2023

ip <> peer ids
21291703847634_ pic

@ping-ke
Copy link
Collaborator

ping-ke commented Jan 10, 2024

The es-node uses UDP to broadcast and receive ENR and then puts the received ENRs (nodes) into the disc_v5 routing table. In Discovery. go of es-node, the DiscoveryProcess puts the nodes in the routing table into Peerstore (if there is Peer information retained in the previous startup in esnode_PeerStore_db, it will also be loaded into PeerStore) and then the libp2p will connect nodes in the Peerstore through TCP.
But if a node does not have a public IP and p2p.advertise.ip is not set, then there is no IP in its ENR, so when other nodes receive it and add it to disc v5, it cannot successfully put it into the routing table after receiving its ENR. See the code below.
https://github.com/ethereum/go-ethereum/blob/master/p2p/discover/table.go line#536

func (tab *Table) addSeenNode(n *node) {
	... ...
	if !tab.addIP(b, n.IP()) {
		// Can't add: IP limit reached.
		return
	}

Therefore, for nodes with public IP, ALL nodes can broadcast their ENR to ALL nodes through routing table, so their Peer number is relatively high. Nodes without public IP can only obtain other nodes through the routing table (only contain nodes with public IP), but cannot connect to other nodes which only have private IP, so there is no connection between private IP nodes.
The Addresses currently in the routing table are:

5.9.87.214:30305 
88.99.30.186:30305 
65.109.62.253:30305 
65.109.115.36:30305 
213.133.99.85:30305 
65.109.50.145:30305 
62.72.46.81:30305 

Other nodes:

73.170.40.66:9222    
65.108.234.173:9222  
65.108.236.27:9222   
142.132.208.253:9222 
119.86.43.247:19457  
162.219.87.212:9222 

@qzhodl
Copy link
Collaborator Author

qzhodl commented Jan 10, 2024

65.108.236.27 is one of our AX101 which has a public IP, but why it is in the "other nodes"?

@ping-ke
Copy link
Collaborator

ping-ke commented Jan 12, 2024

This is because it is run in the docker which does not have a public IP in the docker.

@zhang-wenchao
Copy link

I hope that the network problems existing in docker operation will be solved as soon as possible.

@zhang-wenchao
Copy link

Docker may be easy to deploy, but there is an extra layer of networking and storage, could there be a manual on best practices for running in docker.

@zhang-wenchao
Copy link

zhang-wenchao commented Jan 12, 2024

2024-01-12T17:50:00.400644265Z t=2024-01-12T17:50:00+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/9222
2024-01-12T17:50:01.873911699Z t=2024-01-12T17:50:01+0000 lvl=info msg="Storage sync in progress"              progress=43.14% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,107,[email protected] emptyToFill=1,668,839 timeUsed=25h37m25.617s eta=33h46m31.710s
2024-01-12T17:50:10.025421154Z t=2024-01-12T17:50:10+0000 lvl=info msg="Storage sync in progress"              progress=43.14% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,107,[email protected] emptyToFill=1,668,471 timeUsed=25h37m33.769s eta=33h46m31.578s
2024-01-12T17:50:10.056396294Z t=2024-01-12T17:50:10+0000 lvl=info msg="Download range"                        cache=true       start=303,437 end=303,438   blobNumber=0 duration(ms)=318
2024-01-12T17:50:12.087886462Z t=2024-01-12T17:50:12+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/38632
2024-01-12T17:50:15.219109115Z t=2024-01-12T17:50:15+0000 lvl=warn msg="Cannot register peer for sync duties, peer was already registered" peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac
2024-01-12T17:50:15.219201735Z t=2024-01-12T17:50:15+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/38616
2024-01-12T17:50:15.564774568Z t=2024-01-12T17:50:15+0000 lvl=warn msg="Cannot register peer for sync duties, peer was already registered" peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac
2024-01-12T17:50:15.564842311Z t=2024-01-12T17:50:15+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/9222
2024-01-12T17:50:18.079754332Z t=2024-01-12T17:50:18+0000 lvl=info msg="Storage sync in progress"              progress=43.14% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,108,[email protected] emptyToFill=1,668,087 timeUsed=25h37m41.823s eta=33h46m30.855s
2024-01-12T17:50:18.900495848Z t=2024-01-12T17:50:18+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:50:22.423896782Z t=2024-01-12T17:50:22+0000 lvl=info msg="Download range"                        cache=true       start=303,439 end=303,439   blobNumber=0 duration(ms)=342
2024-01-12T17:50:22.782196611Z t=2024-01-12T17:50:22+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac err="i/o deadline reached"
2024-01-12T17:50:26.166050955Z t=2024-01-12T17:50:26+0000 lvl=info msg="Storage sync in progress"              progress=43.15% peerCount=2 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,108,[email protected] emptyToFill=1,667,687 timeUsed=25h37m49.909s eta=33h46m29.698s
2024-01-12T17:50:28.399306801Z t=2024-01-12T17:50:28+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/38632
2024-01-12T17:50:29.664774553Z t=2024-01-12T17:50:29+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:50:32.782555255Z t=2024-01-12T17:50:32+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac err="i/o deadline reached"
2024-01-12T17:50:34.249685231Z t=2024-01-12T17:50:34+0000 lvl=info msg="Storage sync in progress"              progress=43.15% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,108,[email protected] emptyToFill=1,667,303 timeUsed=25h37m57.993s eta=33h46m28.996s
2024-01-12T17:50:35.693801362Z t=2024-01-12T17:50:35+0000 lvl=info msg="Download range"                        cache=true       start=303,440 end=303,440   blobNumber=0 duration(ms)=323
2024-01-12T17:50:39.383319454Z t=2024-01-12T17:50:39+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/9222
2024-01-12T17:50:39.383394820Z t=2024-01-12T17:50:39+0000 lvl=info msg="Cannot remove peer from sync duties, peer was not registered" peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac
2024-01-12T17:50:39.665469059Z t=2024-01-12T17:50:39+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: i/o deadline reached"
2024-01-12T17:50:42.392217539Z t=2024-01-12T17:50:42+0000 lvl=info msg="Storage sync in progress"              progress=43.15% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,109,[email protected] emptyToFill=1,666,919 timeUsed=25h38m6.136s  eta=33h46m28.387s
2024-01-12T17:50:43.721043510Z t=2024-01-12T17:50:43+0000 lvl=info msg="P2P Summary"                           activePeers=1 inbound=0 outbound=1
2024-01-12T17:50:49.229372019Z t=2024-01-12T17:50:49+0000 lvl=info msg="Download range"                        cache=true       start=303,441 end=303,441   blobNumber=0 duration(ms)=904
2024-01-12T17:50:49.667863794Z t=2024-01-12T17:50:49+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: i/o deadline reached"
2024-01-12T17:50:49.843857348Z t=2024-01-12T17:50:49+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: stream reset"
2024-01-12T17:50:49.847787021Z t=2024-01-12T17:50:49+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:50:49.847841858Z t=2024-01-12T17:50:49+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:50:50.488869387Z t=2024-01-12T17:50:50+0000 lvl=info msg="Storage sync in progress"              progress=43.15% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,109,[email protected] emptyToFill=1,666,535 timeUsed=25h38m14.232s eta=33h46m27.706s
2024-01-12T17:50:51.354286429Z t=2024-01-12T17:50:51+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:50:58.518159388Z t=2024-01-12T17:50:58+0000 lvl=info msg="Storage sync in progress"              progress=43.15% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,109,[email protected] emptyToFill=1,666,135 timeUsed=25h38m22.262s eta=33h46m26.471s
2024-01-12T17:51:02.953035456Z t=2024-01-12T17:51:02+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:51:03.312505472Z t=2024-01-12T17:51:03+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: stream reset"
2024-01-12T17:51:03.317473919Z t=2024-01-12T17:51:03+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:51:03.317498777Z t=2024-01-12T17:51:03+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:51:06.055384934Z t=2024-01-12T17:51:06+0000 lvl=info msg="Download range"                        cache=true       start=303,442 end=303,442   blobNumber=0 duration(ms)=1443
2024-01-12T17:51:06.558317096Z t=2024-01-12T17:51:06+0000 lvl=info msg="Storage sync in progress"              progress=43.16% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,110,[email protected] emptyToFill=1,665,751 timeUsed=25h38m30.302s eta=33h46m25.712s
2024-01-12T17:51:09.866063587Z t=2024-01-12T17:51:09+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/38472
2024-01-12T17:51:14.624055950Z t=2024-01-12T17:51:14+0000 lvl=info msg="Storage sync in progress"              progress=43.16% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,110,[email protected] emptyToFill=1,665,383 timeUsed=25h38m38.368s eta=33h46m25.465s
2024-01-12T17:51:18.433080606Z t=2024-01-12T17:51:18+0000 lvl=info msg="Download range"                        cache=true       start=303,443 end=303,444   blobNumber=0 duration(ms)=1486
2024-01-12T17:51:22.684116626Z t=2024-01-12T17:51:22+0000 lvl=info msg="Storage sync in progress"              progress=43.16% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,111,[email protected] emptyToFill=1,664,999 timeUsed=25h38m46.427s eta=33h46m24.736s
2024-01-12T17:51:29.636216241Z t=2024-01-12T17:51:29+0000 lvl=info msg="Download range"                        cache=true       start=303,445 end=303,445   blobNumber=0 duration(ms)=304
2024-01-12T17:51:30.685463994Z t=2024-01-12T17:51:30+0000 lvl=info msg="Storage sync in progress"              progress=43.16% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,111,[email protected] emptyToFill=1,664,615 timeUsed=25h38m54.429s eta=33h46m23.932s
2024-01-12T17:51:36.793252169Z t=2024-01-12T17:51:36+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:51:38.870776599Z t=2024-01-12T17:51:38+0000 lvl=info msg="Storage sync in progress"              progress=43.17% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,111,[email protected] emptyToFill=1,664,247 timeUsed=25h39m2.614s  eta=33h46m23.834s
2024-01-12T17:51:41.986727752Z t=2024-01-12T17:51:41+0000 lvl=info msg="Download range"                        cache=true       start=303,446 end=303,446   blobNumber=0 duration(ms)=326
2024-01-12T17:51:43.721070146Z t=2024-01-12T17:51:43+0000 lvl=info msg="P2P Summary"                           activePeers=1 inbound=1 outbound=0
2024-01-12T17:51:46.794573986Z t=2024-01-12T17:51:46+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:51:46.888256883Z t=2024-01-12T17:51:46+0000 lvl=info msg="Storage sync in progress"              progress=43.17% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,112,[email protected] emptyToFill=1,663,847 timeUsed=25h39m10.632s eta=33h46m22.578s
2024-01-12T17:51:54.305619033Z t=2024-01-12T17:51:54+0000 lvl=info msg="Download range"                        cache=true       start=303,447 end=303,447   blobNumber=0 duration(ms)=321
2024-01-12T17:51:55.124483843Z t=2024-01-12T17:51:55+0000 lvl=info msg="Storage sync in progress"              progress=43.17% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,112,[email protected] emptyToFill=1,663,463 timeUsed=25h39m18.868s eta=33h46m22.076s
2024-01-12T17:51:56.795586226Z t=2024-01-12T17:51:56+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:52:03.356249989Z t=2024-01-12T17:52:03+0000 lvl=info msg="Storage sync in progress"              progress=43.17% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,113,[email protected] emptyToFill=1,663,063 timeUsed=25h39m27.099s eta=33h46m21.092s
2024-01-12T17:52:06.797138667Z t=2024-01-12T17:52:06+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: i/o deadline reached"
2024-01-12T17:52:07.099781408Z t=2024-01-12T17:52:07+0000 lvl=info msg="Download range"                        cache=true       start=303,448 end=303,448   blobNumber=0 duration(ms)=353
2024-01-12T17:52:12.285031088Z t=2024-01-12T17:52:12+0000 lvl=info msg="Storage sync in progress"              progress=43.17% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,113,[email protected] emptyToFill=1,662,679 timeUsed=25h39m36.029s eta=33h46m21.508s
2024-01-12T17:52:16.799234542Z t=2024-01-12T17:52:16+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: i/o deadline reached"
2024-01-12T17:52:19.710365825Z t=2024-01-12T17:52:19+0000 lvl=info msg="Download range"                        cache=true       start=303,449 end=303,449   blobNumber=0 duration(ms)=320
2024-01-12T17:52:20.412659764Z t=2024-01-12T17:52:20+0000 lvl=info msg="Storage sync in progress"              progress=43.18% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,113,[email protected] emptyToFill=1,662,295 timeUsed=25h39m44.156s eta=33h46m20.863s
2024-01-12T17:52:22.595925768Z t=2024-01-12T17:52:22+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: stream reset"
2024-01-12T17:52:22.595969838Z t=2024-01-12T17:52:22+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/38472
2024-01-12T17:52:28.761979118Z t=2024-01-12T17:52:28+0000 lvl=info msg="Storage sync in progress"              progress=43.18% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,114,[email protected] emptyToFill=1,661,895 timeUsed=25h39m52.505s eta=33h46m20.030s
2024-01-12T17:52:29.975860782Z t=2024-01-12T17:52:29+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:52:32.150501090Z t=2024-01-12T17:52:32+0000 lvl=info msg="Download range"                        cache=true       start=303,450 end=303,450   blobNumber=0 duration(ms)=383
2024-01-12T17:52:36.886996767Z t=2024-01-12T17:52:36+0000 lvl=info msg="Storage sync in progress"              progress=43.18% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,114,[email protected] emptyToFill=1,661,511 timeUsed=25h40m0.630s  eta=33h46m19.387s
2024-01-12T17:52:40.272337067Z t=2024-01-12T17:52:40+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:52:40.272404032Z t=2024-01-12T17:52:40+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:52:43.721389014Z t=2024-01-12T17:52:43+0000 lvl=info msg="P2P Summary"                           activePeers=0 inbound=0 outbound=0
2024-01-12T17:52:44.980064553Z t=2024-01-12T17:52:44+0000 lvl=info msg="Download range"                        cache=true       start=303,451 end=303,451   blobNumber=0 duration(ms)=885
2024-01-12T17:52:44.990041022Z t=2024-01-12T17:52:44+0000 lvl=info msg="Storage sync in progress"              progress=43.18% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,114,[email protected] emptyToFill=1,661,127 timeUsed=25h40m8.734s  eta=33h46m18.700s
2024-01-12T17:52:46.888022034Z t=2024-01-12T17:52:46+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:52:53.073611167Z t=2024-01-12T17:52:53+0000 lvl=info msg="Storage sync in progress"              progress=43.19% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,115,[email protected] emptyToFill=1,660,743 timeUsed=25h40m16.817s eta=33h46m17.998s
2024-01-12T17:52:56.740601599Z t=2024-01-12T17:52:56+0000 lvl=info msg="Download range"                        cache=true       start=303,452 end=303,452   blobNumber=0 duration(ms)=322
2024-01-12T17:52:58.193098109Z t=2024-01-12T17:52:58+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:52:58.493226887Z t=2024-01-12T17:52:58+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: stream reset"
2024-01-12T17:52:58.493309051Z t=2024-01-12T17:52:58+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:52:58.493531277Z t=2024-01-12T17:52:58+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:53:01.184664104Z t=2024-01-12T17:53:01+0000 lvl=info msg="Storage sync in progress"              progress=43.19% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,115,[email protected] emptyToFill=1,660,359 timeUsed=25h40m24.928s eta=33h46m17.319s
2024-01-12T17:53:05.885451733Z t=2024-01-12T17:53:05+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:53:09.598567259Z t=2024-01-12T17:53:09+0000 lvl=info msg="Storage sync in progress"              progress=43.19% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,116,[email protected] emptyToFill=1,659,959 timeUsed=25h40m33.342s eta=33h46m16.573s
2024-01-12T17:53:09.736228586Z t=2024-01-12T17:53:09+0000 lvl=info msg="Download range"                        cache=true       start=303,453 end=303,453   blobNumber=0 duration(ms)=951
2024-01-12T17:53:16.862039444Z t=2024-01-12T17:53:16+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:53:17.163514137Z t=2024-01-12T17:53:17+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: stream reset"
2024-01-12T17:53:17.167114832Z t=2024-01-12T17:53:17+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:53:17.167160835Z t=2024-01-12T17:53:17+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:53:17.647930761Z t=2024-01-12T17:53:17+0000 lvl=info msg="Storage sync in progress"              progress=43.19% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,116,[email protected] emptyToFill=1,659,575 timeUsed=25h40m41.391s eta=33h46m15.821s
2024-01-12T17:53:18.723100885Z t=2024-01-12T17:53:18+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/47182
2024-01-12T17:53:21.763303362Z t=2024-01-12T17:53:21+0000 lvl=info msg="Download range"                        cache=true       start=303,454 end=303,454   blobNumber=0 duration(ms)=325
2024-01-12T17:53:25.958889379Z t=2024-01-12T17:53:25+0000 lvl=info msg="Storage sync in progress"              progress=43.20% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,116,[email protected] emptyToFill=1,659,191 timeUsed=25h40m49.702s eta=33h46m15.412s
2024-01-12T17:53:28.779153197Z t=2024-01-12T17:53:28+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:53:28.783479290Z t=2024-01-12T17:53:28+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/47182
2024-01-12T17:53:28.783681919Z t=2024-01-12T17:53:28+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:53:34.031808543Z t=2024-01-12T17:53:34+0000 lvl=info msg="Storage sync in progress"              progress=43.20% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,117,[email protected] emptyToFill=1,658,807 timeUsed=25h40m57.775s eta=33h46m14.688s
2024-01-12T17:53:38.439274141Z t=2024-01-12T17:53:38+0000 lvl=info msg="error getting latest header"           err="context deadline exceeded"
2024-01-12T17:53:42.093371457Z t=2024-01-12T17:53:42+0000 lvl=info msg="Storage sync in progress"              progress=43.20% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,117,[email protected] emptyToFill=1,658,407 timeUsed=25h41m5.837s  eta=33h46m13.472s
2024-01-12T17:53:43.721107977Z t=2024-01-12T17:53:43+0000 lvl=info msg="P2P Summary"                           activePeers=0 inbound=0 outbound=0
2024-01-12T17:53:44.421011633Z t=2024-01-12T17:53:44+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:53:46.335825679Z t=2024-01-12T17:53:46+0000 lvl=warn msg="Cannot register peer for sync duties, peer was already registered" peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse
2024-01-12T17:53:46.335921039Z t=2024-01-12T17:53:46+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/44180
2024-01-12T17:53:50.232356125Z t=2024-01-12T17:53:50+0000 lvl=info msg="Storage sync in progress"              progress=43.20% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,118,[email protected] emptyToFill=1,658,039 timeUsed=25h41m13.976s eta=33h46m13.302s
2024-01-12T17:53:54.845321022Z t=2024-01-12T17:53:54+0000 lvl=info msg="Download range"                        cache=true       start=303,455 end=303,457   blobNumber=0 duration(ms)=1195
2024-01-12T17:53:57.858384740Z t=2024-01-12T17:53:57+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:53:58.279228971Z t=2024-01-12T17:53:58+0000 lvl=info msg="Storage sync in progress"              progress=43.20% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,118,[email protected] emptyToFill=1,657,655 timeUsed=25h41m22.023s eta=33h46m12.553s
2024-01-12T17:54:06.388475227Z t=2024-01-12T17:54:06+0000 lvl=info msg="Storage sync in progress"              progress=43.21% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,118,[email protected] emptyToFill=1,657,271 timeUsed=25h41m30.132s eta=33h46m11.859s
2024-01-12T17:54:07.838861094Z t=2024-01-12T17:54:07+0000 lvl=info msg="Download range"                        cache=true       start=303,458 end=303,458   blobNumber=0 duration(ms)=1020
2024-01-12T17:54:07.859647694Z t=2024-01-12T17:54:07+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:54:09.552508968Z t=2024-01-12T17:54:09+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:54:14.466485957Z t=2024-01-12T17:54:14+0000 lvl=info msg="Storage sync in progress"              progress=43.21% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,119,[email protected] emptyToFill=1,656,839 timeUsed=25h41m38.210s eta=33h46m9.720s
2024-01-12T17:54:14.844703652Z t=2024-01-12T17:54:14+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:54:14.848619820Z t=2024-01-12T17:54:14+0000 lvl=info msg="Cannot remove peer from sync duties, peer was not registered" peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse
2024-01-12T17:54:14.848692264Z t=2024-01-12T17:54:14+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/44180
2024-01-12T17:54:17.919052532Z t=2024-01-12T17:54:17+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:54:19.491598242Z t=2024-01-12T17:54:19+0000 lvl=info msg="Download range"                        cache=true       start=303,459 end=303,459   blobNumber=0 duration(ms)=336
2024-01-12T17:54:23.222682988Z t=2024-01-12T17:54:23+0000 lvl=info msg="Storage sync in progress"              progress=43.21% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,119,[email protected] emptyToFill=1,656,439 timeUsed=25h41m46.966s eta=33h46m9.420s
2024-01-12T17:54:27.907019177Z t=2024-01-12T17:54:27+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:54:27.910307361Z t=2024-01-12T17:54:27+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:54:27.910354898Z t=2024-01-12T17:54:27+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:54:31.699675559Z t=2024-01-12T17:54:31+0000 lvl=info msg="Storage sync in progress"              progress=43.21% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,120,[email protected] emptyToFill=1,656,039 timeUsed=25h41m55.443s eta=33h46m8.744s
2024-01-12T17:54:39.440274731Z t=2024-01-12T17:54:39+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:54:39.606682604Z t=2024-01-12T17:54:39+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:54:39.606899937Z t=2024-01-12T17:54:39+0000 lvl=warn msg="Cannot register peer for sync duties, peer was already registered" peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse
2024-01-12T17:54:39.865883531Z t=2024-01-12T17:54:39+0000 lvl=info msg="Storage sync in progress"              progress=43.22% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,120,[email protected] emptyToFill=1,655,655 timeUsed=25h42m3.609s  eta=33h46m8.144s
2024-01-12T17:54:43.721657216Z t=2024-01-12T17:54:43+0000 lvl=info msg="P2P Summary"                           activePeers=1 inbound=0 outbound=1

@qzhodl
Copy link
Collaborator Author

qzhodl commented Jan 13, 2024

2024-01-12T17:50:00.400644265Z t=2024-01-12T17:50:00+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/9222
2024-01-12T17:50:01.873911699Z t=2024-01-12T17:50:01+0000 lvl=info msg="Storage sync in progress"              progress=43.14% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,107,[email protected] emptyToFill=1,668,839 timeUsed=25h37m25.617s eta=33h46m31.710s
2024-01-12T17:50:10.025421154Z t=2024-01-12T17:50:10+0000 lvl=info msg="Storage sync in progress"              progress=43.14% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,107,[email protected] emptyToFill=1,668,471 timeUsed=25h37m33.769s eta=33h46m31.578s
2024-01-12T17:50:10.056396294Z t=2024-01-12T17:50:10+0000 lvl=info msg="Download range"                        cache=true       start=303,437 end=303,438   blobNumber=0 duration(ms)=318
2024-01-12T17:50:12.087886462Z t=2024-01-12T17:50:12+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/38632
2024-01-12T17:50:15.219109115Z t=2024-01-12T17:50:15+0000 lvl=warn msg="Cannot register peer for sync duties, peer was already registered" peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac
2024-01-12T17:50:15.219201735Z t=2024-01-12T17:50:15+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/38616
2024-01-12T17:50:15.564774568Z t=2024-01-12T17:50:15+0000 lvl=warn msg="Cannot register peer for sync duties, peer was already registered" peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac
2024-01-12T17:50:15.564842311Z t=2024-01-12T17:50:15+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/9222
2024-01-12T17:50:18.079754332Z t=2024-01-12T17:50:18+0000 lvl=info msg="Storage sync in progress"              progress=43.14% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,108,[email protected] emptyToFill=1,668,087 timeUsed=25h37m41.823s eta=33h46m30.855s
2024-01-12T17:50:18.900495848Z t=2024-01-12T17:50:18+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:50:22.423896782Z t=2024-01-12T17:50:22+0000 lvl=info msg="Download range"                        cache=true       start=303,439 end=303,439   blobNumber=0 duration(ms)=342
2024-01-12T17:50:22.782196611Z t=2024-01-12T17:50:22+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac err="i/o deadline reached"
2024-01-12T17:50:26.166050955Z t=2024-01-12T17:50:26+0000 lvl=info msg="Storage sync in progress"              progress=43.15% peerCount=2 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,108,[email protected] emptyToFill=1,667,687 timeUsed=25h37m49.909s eta=33h46m29.698s
2024-01-12T17:50:28.399306801Z t=2024-01-12T17:50:28+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/38632
2024-01-12T17:50:29.664774553Z t=2024-01-12T17:50:29+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:50:32.782555255Z t=2024-01-12T17:50:32+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac err="i/o deadline reached"
2024-01-12T17:50:34.249685231Z t=2024-01-12T17:50:34+0000 lvl=info msg="Storage sync in progress"              progress=43.15% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,108,[email protected] emptyToFill=1,667,303 timeUsed=25h37m57.993s eta=33h46m28.996s
2024-01-12T17:50:35.693801362Z t=2024-01-12T17:50:35+0000 lvl=info msg="Download range"                        cache=true       start=303,440 end=303,440   blobNumber=0 duration(ms)=323
2024-01-12T17:50:39.383319454Z t=2024-01-12T17:50:39+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac addr=/ip4/65.109.115.36/tcp/9222
2024-01-12T17:50:39.383394820Z t=2024-01-12T17:50:39+0000 lvl=info msg="Cannot remove peer from sync duties, peer was not registered" peer=16Uiu2HAmGomLzB1k1PXGx2SZUNsgJNUxXYj9R986orRXsTuDr7Ac
2024-01-12T17:50:39.665469059Z t=2024-01-12T17:50:39+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: i/o deadline reached"
2024-01-12T17:50:42.392217539Z t=2024-01-12T17:50:42+0000 lvl=info msg="Storage sync in progress"              progress=43.15% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,109,[email protected] emptyToFill=1,666,919 timeUsed=25h38m6.136s  eta=33h46m28.387s
2024-01-12T17:50:43.721043510Z t=2024-01-12T17:50:43+0000 lvl=info msg="P2P Summary"                           activePeers=1 inbound=0 outbound=1
2024-01-12T17:50:49.229372019Z t=2024-01-12T17:50:49+0000 lvl=info msg="Download range"                        cache=true       start=303,441 end=303,441   blobNumber=0 duration(ms)=904
2024-01-12T17:50:49.667863794Z t=2024-01-12T17:50:49+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: i/o deadline reached"
2024-01-12T17:50:49.843857348Z t=2024-01-12T17:50:49+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: stream reset"
2024-01-12T17:50:49.847787021Z t=2024-01-12T17:50:49+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:50:49.847841858Z t=2024-01-12T17:50:49+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:50:50.488869387Z t=2024-01-12T17:50:50+0000 lvl=info msg="Storage sync in progress"              progress=43.15% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,109,[email protected] emptyToFill=1,666,535 timeUsed=25h38m14.232s eta=33h46m27.706s
2024-01-12T17:50:51.354286429Z t=2024-01-12T17:50:51+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:50:58.518159388Z t=2024-01-12T17:50:58+0000 lvl=info msg="Storage sync in progress"              progress=43.15% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,109,[email protected] emptyToFill=1,666,135 timeUsed=25h38m22.262s eta=33h46m26.471s
2024-01-12T17:51:02.953035456Z t=2024-01-12T17:51:02+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:51:03.312505472Z t=2024-01-12T17:51:03+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: stream reset"
2024-01-12T17:51:03.317473919Z t=2024-01-12T17:51:03+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:51:03.317498777Z t=2024-01-12T17:51:03+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:51:06.055384934Z t=2024-01-12T17:51:06+0000 lvl=info msg="Download range"                        cache=true       start=303,442 end=303,442   blobNumber=0 duration(ms)=1443
2024-01-12T17:51:06.558317096Z t=2024-01-12T17:51:06+0000 lvl=info msg="Storage sync in progress"              progress=43.16% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,110,[email protected] emptyToFill=1,665,751 timeUsed=25h38m30.302s eta=33h46m25.712s
2024-01-12T17:51:09.866063587Z t=2024-01-12T17:51:09+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/38472
2024-01-12T17:51:14.624055950Z t=2024-01-12T17:51:14+0000 lvl=info msg="Storage sync in progress"              progress=43.16% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,110,[email protected] emptyToFill=1,665,383 timeUsed=25h38m38.368s eta=33h46m25.465s
2024-01-12T17:51:18.433080606Z t=2024-01-12T17:51:18+0000 lvl=info msg="Download range"                        cache=true       start=303,443 end=303,444   blobNumber=0 duration(ms)=1486
2024-01-12T17:51:22.684116626Z t=2024-01-12T17:51:22+0000 lvl=info msg="Storage sync in progress"              progress=43.16% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,111,[email protected] emptyToFill=1,664,999 timeUsed=25h38m46.427s eta=33h46m24.736s
2024-01-12T17:51:29.636216241Z t=2024-01-12T17:51:29+0000 lvl=info msg="Download range"                        cache=true       start=303,445 end=303,445   blobNumber=0 duration(ms)=304
2024-01-12T17:51:30.685463994Z t=2024-01-12T17:51:30+0000 lvl=info msg="Storage sync in progress"              progress=43.16% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,111,[email protected] emptyToFill=1,664,615 timeUsed=25h38m54.429s eta=33h46m23.932s
2024-01-12T17:51:36.793252169Z t=2024-01-12T17:51:36+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:51:38.870776599Z t=2024-01-12T17:51:38+0000 lvl=info msg="Storage sync in progress"              progress=43.17% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,111,[email protected] emptyToFill=1,664,247 timeUsed=25h39m2.614s  eta=33h46m23.834s
2024-01-12T17:51:41.986727752Z t=2024-01-12T17:51:41+0000 lvl=info msg="Download range"                        cache=true       start=303,446 end=303,446   blobNumber=0 duration(ms)=326
2024-01-12T17:51:43.721070146Z t=2024-01-12T17:51:43+0000 lvl=info msg="P2P Summary"                           activePeers=1 inbound=1 outbound=0
2024-01-12T17:51:46.794573986Z t=2024-01-12T17:51:46+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:51:46.888256883Z t=2024-01-12T17:51:46+0000 lvl=info msg="Storage sync in progress"              progress=43.17% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,112,[email protected] emptyToFill=1,663,847 timeUsed=25h39m10.632s eta=33h46m22.578s
2024-01-12T17:51:54.305619033Z t=2024-01-12T17:51:54+0000 lvl=info msg="Download range"                        cache=true       start=303,447 end=303,447   blobNumber=0 duration(ms)=321
2024-01-12T17:51:55.124483843Z t=2024-01-12T17:51:55+0000 lvl=info msg="Storage sync in progress"              progress=43.17% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,112,[email protected] emptyToFill=1,663,463 timeUsed=25h39m18.868s eta=33h46m22.076s
2024-01-12T17:51:56.795586226Z t=2024-01-12T17:51:56+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:52:03.356249989Z t=2024-01-12T17:52:03+0000 lvl=info msg="Storage sync in progress"              progress=43.17% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,113,[email protected] emptyToFill=1,663,063 timeUsed=25h39m27.099s eta=33h46m21.092s
2024-01-12T17:52:06.797138667Z t=2024-01-12T17:52:06+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: i/o deadline reached"
2024-01-12T17:52:07.099781408Z t=2024-01-12T17:52:07+0000 lvl=info msg="Download range"                        cache=true       start=303,448 end=303,448   blobNumber=0 duration(ms)=353
2024-01-12T17:52:12.285031088Z t=2024-01-12T17:52:12+0000 lvl=info msg="Storage sync in progress"              progress=43.17% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,113,[email protected] emptyToFill=1,662,679 timeUsed=25h39m36.029s eta=33h46m21.508s
2024-01-12T17:52:16.799234542Z t=2024-01-12T17:52:16+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: i/o deadline reached"
2024-01-12T17:52:19.710365825Z t=2024-01-12T17:52:19+0000 lvl=info msg="Download range"                        cache=true       start=303,449 end=303,449   blobNumber=0 duration(ms)=320
2024-01-12T17:52:20.412659764Z t=2024-01-12T17:52:20+0000 lvl=info msg="Storage sync in progress"              progress=43.18% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,113,[email protected] emptyToFill=1,662,295 timeUsed=25h39m44.156s eta=33h46m20.863s
2024-01-12T17:52:22.595925768Z t=2024-01-12T17:52:22+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: stream reset"
2024-01-12T17:52:22.595969838Z t=2024-01-12T17:52:22+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/38472
2024-01-12T17:52:28.761979118Z t=2024-01-12T17:52:28+0000 lvl=info msg="Storage sync in progress"              progress=43.18% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,114,[email protected] emptyToFill=1,661,895 timeUsed=25h39m52.505s eta=33h46m20.030s
2024-01-12T17:52:29.975860782Z t=2024-01-12T17:52:29+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:52:32.150501090Z t=2024-01-12T17:52:32+0000 lvl=info msg="Download range"                        cache=true       start=303,450 end=303,450   blobNumber=0 duration(ms)=383
2024-01-12T17:52:36.886996767Z t=2024-01-12T17:52:36+0000 lvl=info msg="Storage sync in progress"              progress=43.18% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,114,[email protected] emptyToFill=1,661,511 timeUsed=25h40m0.630s  eta=33h46m19.387s
2024-01-12T17:52:40.272337067Z t=2024-01-12T17:52:40+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:52:40.272404032Z t=2024-01-12T17:52:40+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:52:43.721389014Z t=2024-01-12T17:52:43+0000 lvl=info msg="P2P Summary"                           activePeers=0 inbound=0 outbound=0
2024-01-12T17:52:44.980064553Z t=2024-01-12T17:52:44+0000 lvl=info msg="Download range"                        cache=true       start=303,451 end=303,451   blobNumber=0 duration(ms)=885
2024-01-12T17:52:44.990041022Z t=2024-01-12T17:52:44+0000 lvl=info msg="Storage sync in progress"              progress=43.18% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,114,[email protected] emptyToFill=1,661,127 timeUsed=25h40m8.734s  eta=33h46m18.700s
2024-01-12T17:52:46.888022034Z t=2024-01-12T17:52:46+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:52:53.073611167Z t=2024-01-12T17:52:53+0000 lvl=info msg="Storage sync in progress"              progress=43.19% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,115,[email protected] emptyToFill=1,660,743 timeUsed=25h40m16.817s eta=33h46m17.998s
2024-01-12T17:52:56.740601599Z t=2024-01-12T17:52:56+0000 lvl=info msg="Download range"                        cache=true       start=303,452 end=303,452   blobNumber=0 duration(ms)=322
2024-01-12T17:52:58.193098109Z t=2024-01-12T17:52:58+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:52:58.493226887Z t=2024-01-12T17:52:58+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: stream reset"
2024-01-12T17:52:58.493309051Z t=2024-01-12T17:52:58+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:52:58.493531277Z t=2024-01-12T17:52:58+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:53:01.184664104Z t=2024-01-12T17:53:01+0000 lvl=info msg="Storage sync in progress"              progress=43.19% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,115,[email protected] emptyToFill=1,660,359 timeUsed=25h40m24.928s eta=33h46m17.319s
2024-01-12T17:53:05.885451733Z t=2024-01-12T17:53:05+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:53:09.598567259Z t=2024-01-12T17:53:09+0000 lvl=info msg="Storage sync in progress"              progress=43.19% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,116,[email protected] emptyToFill=1,659,959 timeUsed=25h40m33.342s eta=33h46m16.573s
2024-01-12T17:53:09.736228586Z t=2024-01-12T17:53:09+0000 lvl=info msg="Download range"                        cache=true       start=303,453 end=303,453   blobNumber=0 duration(ms)=951
2024-01-12T17:53:16.862039444Z t=2024-01-12T17:53:16+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:53:17.163514137Z t=2024-01-12T17:53:17+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="failed to read result part of response: stream reset"
2024-01-12T17:53:17.167114832Z t=2024-01-12T17:53:17+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:53:17.167160835Z t=2024-01-12T17:53:17+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:53:17.647930761Z t=2024-01-12T17:53:17+0000 lvl=info msg="Storage sync in progress"              progress=43.19% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,116,[email protected] emptyToFill=1,659,575 timeUsed=25h40m41.391s eta=33h46m15.821s
2024-01-12T17:53:18.723100885Z t=2024-01-12T17:53:18+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/47182
2024-01-12T17:53:21.763303362Z t=2024-01-12T17:53:21+0000 lvl=info msg="Download range"                        cache=true       start=303,454 end=303,454   blobNumber=0 duration(ms)=325
2024-01-12T17:53:25.958889379Z t=2024-01-12T17:53:25+0000 lvl=info msg="Storage sync in progress"              progress=43.20% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,116,[email protected] emptyToFill=1,659,191 timeUsed=25h40m49.702s eta=33h46m15.412s
2024-01-12T17:53:28.779153197Z t=2024-01-12T17:53:28+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:53:28.783479290Z t=2024-01-12T17:53:28+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/47182
2024-01-12T17:53:28.783681919Z t=2024-01-12T17:53:28+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:53:34.031808543Z t=2024-01-12T17:53:34+0000 lvl=info msg="Storage sync in progress"              progress=43.20% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,117,[email protected] emptyToFill=1,658,807 timeUsed=25h40m57.775s eta=33h46m14.688s
2024-01-12T17:53:38.439274141Z t=2024-01-12T17:53:38+0000 lvl=info msg="error getting latest header"           err="context deadline exceeded"
2024-01-12T17:53:42.093371457Z t=2024-01-12T17:53:42+0000 lvl=info msg="Storage sync in progress"              progress=43.20% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,117,[email protected] emptyToFill=1,658,407 timeUsed=25h41m5.837s  eta=33h46m13.472s
2024-01-12T17:53:43.721107977Z t=2024-01-12T17:53:43+0000 lvl=info msg="P2P Summary"                           activePeers=0 inbound=0 outbound=0
2024-01-12T17:53:44.421011633Z t=2024-01-12T17:53:44+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:53:46.335825679Z t=2024-01-12T17:53:46+0000 lvl=warn msg="Cannot register peer for sync duties, peer was already registered" peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse
2024-01-12T17:53:46.335921039Z t=2024-01-12T17:53:46+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/44180
2024-01-12T17:53:50.232356125Z t=2024-01-12T17:53:50+0000 lvl=info msg="Storage sync in progress"              progress=43.20% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,118,[email protected] emptyToFill=1,658,039 timeUsed=25h41m13.976s eta=33h46m13.302s
2024-01-12T17:53:54.845321022Z t=2024-01-12T17:53:54+0000 lvl=info msg="Download range"                        cache=true       start=303,455 end=303,457   blobNumber=0 duration(ms)=1195
2024-01-12T17:53:57.858384740Z t=2024-01-12T17:53:57+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:53:58.279228971Z t=2024-01-12T17:53:58+0000 lvl=info msg="Storage sync in progress"              progress=43.20% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,118,[email protected] emptyToFill=1,657,655 timeUsed=25h41m22.023s eta=33h46m12.553s
2024-01-12T17:54:06.388475227Z t=2024-01-12T17:54:06+0000 lvl=info msg="Storage sync in progress"              progress=43.21% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,118,[email protected] emptyToFill=1,657,271 timeUsed=25h41m30.132s eta=33h46m11.859s
2024-01-12T17:54:07.838861094Z t=2024-01-12T17:54:07+0000 lvl=info msg="Download range"                        cache=true       start=303,458 end=303,458   blobNumber=0 duration(ms)=1020
2024-01-12T17:54:07.859647694Z t=2024-01-12T17:54:07+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="i/o deadline reached"
2024-01-12T17:54:09.552508968Z t=2024-01-12T17:54:09+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:54:14.466485957Z t=2024-01-12T17:54:14+0000 lvl=info msg="Storage sync in progress"              progress=43.21% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,119,[email protected] emptyToFill=1,656,839 timeUsed=25h41m38.210s eta=33h46m9.720s
2024-01-12T17:54:14.844703652Z t=2024-01-12T17:54:14+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:54:14.848619820Z t=2024-01-12T17:54:14+0000 lvl=info msg="Cannot remove peer from sync duties, peer was not registered" peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse
2024-01-12T17:54:14.848692264Z t=2024-01-12T17:54:14+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/44180
2024-01-12T17:54:17.919052532Z t=2024-01-12T17:54:17+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:54:19.491598242Z t=2024-01-12T17:54:19+0000 lvl=info msg="Download range"                        cache=true       start=303,459 end=303,459   blobNumber=0 duration(ms)=336
2024-01-12T17:54:23.222682988Z t=2024-01-12T17:54:23+0000 lvl=info msg="Storage sync in progress"              progress=43.21% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,119,[email protected] emptyToFill=1,656,439 timeUsed=25h41m46.966s eta=33h46m9.420s
2024-01-12T17:54:27.907019177Z t=2024-01-12T17:54:27+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="stream reset"
2024-01-12T17:54:27.910307361Z t=2024-01-12T17:54:27+0000 lvl=info msg="Disconnected from peer"                peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:54:27.910354898Z t=2024-01-12T17:54:27+0000 lvl=warn msg="Failed to request blobs"               peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse err="context canceled"
2024-01-12T17:54:31.699675559Z t=2024-01-12T17:54:31+0000 lvl=info msg="Storage sync in progress"              progress=43.21% peerCount=0 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,120,[email protected] emptyToFill=1,656,039 timeUsed=25h41m55.443s eta=33h46m8.744s
2024-01-12T17:54:39.440274731Z t=2024-01-12T17:54:39+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:54:39.606682604Z t=2024-01-12T17:54:39+0000 lvl=info msg="Connected to peer"                     peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse addr=/ip4/213.133.99.85/tcp/9222
2024-01-12T17:54:39.606899937Z t=2024-01-12T17:54:39+0000 lvl=warn msg="Cannot register peer for sync duties, peer was already registered" peer=16Uiu2HAkym6S9pc26eLeoKkao1z4EAoKXY8ysV7n9Am4kXoWAjse
2024-01-12T17:54:39.865883531Z t=2024-01-12T17:54:39+0000 lvl=info msg="Storage sync in progress"              progress=43.22% peerCount=1 syncTasksRemain=1@32 blobsSynced=130,[email protected] blobsToSync=7,870,985 fillTasksRemain=6 emptyFilled=7,120,[email protected] emptyToFill=1,655,655 timeUsed=25h42m3.609s  eta=33h46m8.144s
2024-01-12T17:54:43.721657216Z t=2024-01-12T17:54:43+0000 lvl=info msg="P2P Summary"                           activePeers=1 inbound=0 outbound=1

It is weird that the logs show you can only connect 1 peer, are you following the https://github.com/ethstorage/es-node/blob/main/GUIDE.md here to setup? maybe you can try VPN if you are based in China

@zhang-wenchao
Copy link

@qzhodl Can you please tell me how to use proxy on node, VPN We are going to use rpc and VPN will proxy all the traffic.

@qzhodl
Copy link
Collaborator Author

qzhodl commented Jan 15, 2024

I'm not sure if I've understood your question correctly. Are you asking how to set up the VPN to proxy only the es-node traffic, rather than all the traffic on your machine? I believe the ability to do this depends on the specific VPN software

@zhang-wenchao
Copy link

I don't really want to use a VPN, I also need the local network to use rpc and I already have a socks5 proxy available locally. how do I synchronize using proxy.

@ping-ke
Copy link
Collaborator

ping-ke commented Jan 16, 2024

from the log, the peer count drops to 0, it looks like it was caused by a network issue, a remote peer connected to it and then disconnected in a short time. and during the connection, the peer cannot successfully return the requested blobs in time.

@ping-ke
Copy link
Collaborator

ping-ke commented Jan 16, 2024

I tried to run the es-node in our office in Beijing without a VPN. The node can join the network and the discovery with UDP and P2P with TCP work well. Then I disconnect my computer, and after that, the es-node P2P peers disconnect and the peer count drops to 0 and the routing table in the discovery also be clear.
After I rejoin the computer to the network, the UDP discovery would recover in about 20 minutes. and TCP connection would recover also after discovery.

@qzhodl
Copy link
Collaborator Author

qzhodl commented Jan 17, 2024

I think the top priority is that we can support serving the data to other node when using docker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

3 participants