ProvideMany: high memory usage when providing tens of millions of CIDs

[ajnavarro]

When using BatchProviding, we are not really batching, but sending all the CIDs at the same time to the Router implementing ProvideMany.

To avoid collateral problems, we should actually batch the calls to ProvideMany.

This will help with Reframe Router implementation (https://github.com/ipfs/go-delegated-routing) to avoid huge JSON payloads sent to the server.

We need to find good defaults to still keep FullRT DHT implementation with good performance numbers.

That is like 1/10th of the memory spike observed, we are still searching for other possible problems.

[welcome]

Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
In the meantime, please double-check that you have provided all the necessary information to make this process easy! Any information that can help save additional round trips is useful! We currently aim to give initial feedback within two business days. If this does not happen, feel free to leave a comment.
Please keep an eye on how this issue will be labeled, as labels give an overview of priorities, assignments and additional actions requested by the maintainers:

• "Priority" labels will show how urgent this is for the team.
• "Status" labels will show if this is ready to be worked on, blocked, or in progress.
• "Need" labels will indicate if additional input or analysis is required.

Finally, remember to use https://discuss.ipfs.io if you just need general support.

[ajnavarro]

CC @ischasny

[BigLep]

2022-10-18 conversation: we aren't aware of this being a blocker at the moment so not prioritizing currently, but feedback welcome if this needs to be moved up sooner.

[ischasny]

@ajnavarro as per our discussion, might be a good idea to chunk up the CIDs snapshot into smaller pieces so that at least we don't squeeze all of them into a single HTTP request. That is problematic on larger nodes (like web3 storage). Snapshots don't get reprovided even with high router timeouts. Maybe we can do that only for reframe router initially? That should save us some memory on both sending and receiving side. Wdyt?

[ajnavarro]

Yeah, won't be the final solution, but will help in providing over HTTP.

[ischasny]

Great! Would you guys be up for taking it into the next release? Should be simple to do and would unblock us too.

[ajnavarro]

Related issue: ipfs/go-delegated-routing#55