Respond to chunk requests once (per some time interval)

[birchmd]

Nodes will request chunk parts they are missing when they get a block which includes a chunk they do not yet have (either have their part for or have reconstructed if it is for a shard they track). These requests are re-sent relatively frequently to ensure they are delivered and responded to. However, presently nodes will always respond to these requests, even if they have responded in the past. This can lead to redundant messages being sent over the network and could potentially contribute to congestion which is slowing down the network.

In this issue we want to add logic to node such they they will only respond to a request once per some time period (which can then be tuned for performance).

[bowenwang1996]

I think it may be better to do this on the network level, for all requests that require responses.

[birchmd]

Update on this issue:

We tried not accepting duplicate messages at the network level, remembering which we had seen before using the sha256 hash of the message (commit). But the hashing of messages introduced more overhead than ignoring duplicates saved.

We tried storing the raw message bytes as well (commit). This seemed to improve performance a little bit, though made the memory use much higher, probably too high to be an acceptable solution for production.

We tried using xxHash instead of sha256 since it is a faster hash function, but this also didn't seem to help performance very much. However, this might still be worth moving to production as a simple DOS prevention measure.

[birchmd]

Update: upgrading to the latest actix/tokio versions (see #3868 and #3869) seems to have a much greater impact on performance (may even allow sustaining 4k tps on 8 shards). The possibility of DOS attacks via flooding nodes with messages (especially since the p2p network is fully open) still exists, but that can be addressed as a general stability/security fix, rather than a sharding performance improvement. We also have an issue which is likely to reduce the amount of network traffic generated by (honest) non-validating nodes (see #3890).

Given the above, I am closing this issue.

[bowenwang1996]

I don't think we should close this issue given that it is not done and we still need to do it.

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months.
It will be closed in 7 days if no further activity occurs.
Thank you for your contributions.