Peer discovery issue - Error sending status request

[phahulin]

I'm running:

• Which Parity version?: 1.10.6
• Which operating system?: Linux
• How installed?: via binaries
• Are you fully synchronized?: -
• Which network are you connected to?: sokol
• Did you try to restart the node?: yes

we run a poa-based network and I noticed that recently new nodes joining the network have very low peers count. This might be related to our security upgrade from 1.9.2 to 1.10.6.

"Old" nodes (the ones launched some time ago) maintain high peers count (30-40). However, when a new node joins the network peers count stays low (3-5). It appears that some of the old nodes no longer can be synced from while others still can, I don't see what differentiates them, all old nodes are in sync and look similar.

I did the following test: started parity locally on my laptop with correct genesis file but without any peers:

./parity --chain sokol-spec.json --jsonrpc-apis parity,parity_set --logging sync=trace,network=trace --base-path pdata 2>log.txt

Waited for 0/25 peers message to appear in logs. Then I added one of the old nodes as reserved peer

curl --data '{"method":"parity_addReservedPeer","params":["enode://0171db265a569636372566e86cb7a69306fe5c15a8e2ed5bed4010012fa1d146ae4918a688cf1bd3fd98e8c2d5c3705d68ff941c88ab974ff52c7fc8606ce2f8@1.2.3.4:30303"],"id":1,"jsonrpc":"2.0"}' -H "Content-Type: application/json" -X POST localhost:8545
{"jsonrpc":"2.0","result":true,"id":1}

(I changed the IP address here for security purposes). And still there were 0/25 peers.
Here is the log, I see an error message

Error sending status request: Error(Expired, State { next_error: None, backtrace: None })

[Tbaut]

Duplicates #7531
Please run the node with --no-discovery or upgrade to 1.11.X

[phahulin]

I updated parity to 1.11.4 both locally and on the remote node, but get the same error. Here's the new log

[roninkaizen]

as you open each node individually and delete the nodes.json from the chains directory
and recheck, it could work again,
-->tested in distributed ubuntu 18.04 environment with poa-standard.environment as discribed within the wiki with 4 nodes over 4 hours parallel connect to main.chain with version parity_1.10.7<--

[phahulin]

I can't test it in full because I don't have access to majority of nodes of our network. I'm also not sure that deleting nodes.json files on old nodes is a good idea, since those might be the only correct ones.

I upgraded two nodes to 1.11.4. On one of these two, I completely deleted parity data folder (including nodes.json) and resynced it, with a fresh enode. It now only has a single peer, and it's not the second node.

It'd be good if someone could clarify what does this error (Error sending status request: Error(Expired,...) indicate, it's not clear for me from parity source code.

[Tbaut]

This Error appears here: https://github.com/paritytech/parity/blob/2060ea5de388f7b3b515ba5c2d58425585a8ac1e/ethcore/sync/src/chain/handler.rs#L128-L136

It actually looks like you have connectivity issues as it means that you'll disconnect from the peer. How many nodes do you have on the network?

[phahulin]

In total there are about 60 nodes, we keep a list of 12 of them to use as --reserved-peers for new nodes joining the network and then it usually grows up to 30-40 peers.
I just checked that telnet on 30303 port works for all 12 nodes except 1, so seems like there are 11 available peers in the list.

[Tbaut]

It can take a while to connect to all nodes, but the reserved-peers should definitely connect quickly. Alternatively, you could try to sync with --reserved-only and check the logs to see what happens.

[phahulin]

This is basically what it did - I started with 0 peers and add a single peer as reserved peer. Link to logs are above.

[jam10o-new]

Is this still an issue with the recent discovery fixes? @phahulin lmk if this still persists in the recent stable and you need this reopened

[danihodovic]

I'm seeing this issue on Parity nodes that have been restored from an S3 backup. The node connects to 1 out of 7 peers in total. These nodes only serve as API nodes for dApps and therefore have no credentials attached to them. We're running a proof-of-authority chain.

Procedure:

1. Create a tarfile out of an entire Parity directory. Exclude:

• /opt/parity/password.pwds
• /opt/parity/network/key
• /opt/parity/keys

2. Save the tarfile to S3
3. Boot a new Parity instance, copy the backup into /opt/parity and start Parity.

Parity version:

root@7fc47f247dcc:/opt/parity# /home/parity/bin/parity --version
Parity Ethereum
  version Parity-Ethereum/v2.2.7-stable-b00a21f-20190115/x86_64-linux-gnu/rustc1.31.1

Stacktrace:

2019-01-23 08:15:48 UTC IO Worker #1 DEBUG sync  25 -> Invalid packet 0
2019-01-23 08:15:48 UTC IO Worker #2 DEBUG sync  42 -> Invalid packet 0
2019-01-23 08:15:48 UTC IO Worker #1 DEBUG sync  33 -> Invalid packet 0
2019-01-23 08:15:48 UTC IO Worker #2 DEBUG sync  Error sending status request: Error(Expired, State { next_error: None, backtrace: InternalBacktrace { backtrace: Some(stack backtrace:
   0:     0x55ab294976dd - <no info>
   1:     0x55ab29496092 - <no info>
   2:     0x55ab2907d421 - <no info>
   3:     0x55ab2907d755 - <no info>
   4:     0x55ab289023c2 - <no info>
   5:     0x55ab2871adc8 - <no info>
   6:     0x55ab286f23ee - <no info>
   7:     0x55ab286f26ba - <no info>
   8:     0x55ab28676138 - <no info>
   9:     0x55ab286af94f - <no info>
  10:     0x55ab2868d076 - <no info>
  11:     0x55ab2868195b - <no info>
  12:     0x55ab286fd66f - <no info>
  13:     0x55ab2872464b - <no info>
  14:     0x55ab286e3966 - <no info>
  15:     0x55ab286e3bf6 - <no info>
  16:     0x55ab29521e79 - <no info>
  17:     0x55ab286ebd67 - <no info>
  18:     0x55ab2950c12d - <no info>
  19:     0x55ab294f6ce5 - <no info>
  20:     0x7f1949ea46b9 - <no info>
  21:     0x7f19499c441c - <no info>
  22:                0x0 - <no info>) } })

Config

[parity]
base_path = "/opt/parity"
chain = "chain.json"
mode = "active"

[footprint]
cache_size_db = 777
pruning = "fast"

[network]
nat = "extip:<ip>"
warp = false
reserved_peers = "reserved.txt"
min_peers = 3

[rpc]
apis = ["web3", "eth", "net", "parity"]
interface = "all"
cors = ["*"]

[websockets]
apis = ["web3", "eth", "pubsub", "net"]
interface = "all"
origins = ["all"]

[ipc]
disable = true

[misc]
logging = "own_tx=warn,sync=debug"

[danihodovic]

Update 1:

It turns out that the network key which is responsible for generating the enode was the same as the one from the backed up node. Once I deleted it and it generated a new enode the peer count went up to 2. I'm not sure if that's related to this problem. However it doesn't go beyond 2 and the stacktrace is still being logged.

rm /opt/parity/network/key

Update 2:

I stopped Parity and deleted the below files. After that Parity immediately synced to the reserved peers.

• /opt/parity/chains/<chain>/user_defaults
• /opt/parity/chains/<chain>/network/nodes.json

The content of /opt/parity/chains/<chain>/user_defaults was

{"is_first_launch":false,"pruning":"fast","tracing":false,"fat_db":false,"mode":"active"}

[adria0]

Closing issue due to its stale state.