Excessive sampling requests sent to peer(s)

[jimmygchen]

Description

After inbound rate limiting was disabled on our Lighthouse supernodes, one of our nodes (16Uiu2HAmVFVKtoALhWo3p9mbBYHnhkZLW44qu47V7bgZnzf3dUeq) on peerdas-devnet-1 sent out excessive sampling requests to its peers after a restart, and may have caused the other LH supernode to stop following the head.

lighthouse-geth-1 received 419k DataColumnsByRoot requests in around 3 hours.

grep "Received DataColumnsByRoot Request" beacon.log.5 | awk -F 'peer: ' '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -nr
 419956 16Uiu2HAmVFVKtoALhWo3p9mbBYHnhkZLW44qu47V7bgZnzf3dUeq,

Upon checking the requests sent from lighthouse-geth-2, it looks like the node sent out a lot of requests for the same root:

grep "Sending DataColumnsByRoot Request" beacon.log.* | awk -F 'block_root: ' '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -nr
   3312 0x08fec9837f79366d0dc1f4fecdef88897e14bc7e3b5c2fa7fbdcfad80174867f,
   3093 0x0b4b1dc2bcd0da11452638b02f7a1af8453b4abda07073eab77c53d47820fc36,
   3085 0xd1d92ce4c9ac309b3c3b94a1f933a69b27a1fa6ba8813d81d19f6ac24458d023,
   3065 0xbb18d86b5dd28cf6aef2160ebdf294819da85257f7a7a4aa8956f260be2c3f42,
   2985 0x6cb749baeec5cfbb68c2fc15636d5793e72d51e8e9213d50efb1b1e473baf922,
   2978 0x560ff5701aa12e469a73d2ea97a81bb6a8962cb1b2e9465e3c55234b52b1679d,
   2927 0xb78f5ec2dbe4f841ee980fec0d168aa91c0b1db0a88b6b5a1c8974fd6577e4bd,
   2909 0x5bb0d9ed06121b20544e61ecd7c7eba3dbb0a745e34faa74f83d83bef27d22a7,
   2897 0x88a39eede32b866bf3ef1a08f3f49c0b86553c4b4b022596912f274ab1bbdb3f,

[jimmygchen]

Fixed in #6084