Stream sync: context deadline exceeded

[Frozen]

Describe the bug
{"level":"error","port":"9000","ip":"127.0.0.1","module":"staged stream sync","ShardID":0,"error":"context deadline exceeded","streamID":"QmSAaTjhHWNF42A4xWLkmcwnmEJhatjbqMpv84HBxeNzkt","caller":"/Users/frozen/go/src/github.com/harmony-one/harmony/api/service/stagedstreamsync/syncing.go:520","time":"2024-09-03T13:45:10.258677-04:00","message":"[STAGED_STREAM_SYNC]: getCurrentNumber request failed"}

lib p2p stream being replaced with new one, but message is expected from old stream. Need to understand why new streams are created, while we already have connection.

To Reproduce
Steps to reproduce the behavior:

1. Check out code with dev branch
2. Build
3. run make debug

Expected behavior
This error should never happen because it's localhost and no network issue are possible.

[Frozen]

{"level":"warn","port":"9002","ip":"127.0.0.1","mode":"epoch chain short range","error":"stream removed when doing request","stream":"QmTXwseacFvVFeHfSBSwL3xdAHMyTeesNUu2DJaGuYWZt5","caller":"/Users/frozen/go/src/github.com/harmony-one/harmony/api/service/stagedstreamsync/short_range_helper.go:185","time":"2024-09-03T22:43:13.597391-04:00","message":"[STAGED_STREAM_SYNC]: failed to doGetBlockHashesRequest"}

[GheisMohammadi]

The removal of the P2P stream is expected in this case, as the getCurrentNumber request failed multiple times. When the sync process is unable to retrieve data from the stream after several attempts, the stream is replaced. However, the key issue lies in understanding why the getCurrentNumber request is failing in the first place. This needs further investigation to determine the root cause behind the failed request.

[GheisMohammadi]

The PR #4762 tries to address the issue. It is in progress.

[GheisMohammadi]

Investigations showed the issue was introduced by the latest update to the muxers for the P2P host. Switching localnet to exclusively use Yamux (PR #4764 ) resolved the stream instability problems.