Split provide messages (message size limits)

[willscott]

When providing an array of many items, if the number of items is very large, we should consider splitting the request into multiple smaller messages.

cc @lidel - should there be a maximum in the spec for the number of accepted provided items a single delegated routing call may contain?

[lidel]

Yes, we should always have max sizes as part of specs, just to ensure implementations don't create any footguns.

cc @ajnavarro for visibility, as i know you've been looking at a similar issue in Kubo, where we were sending provide (or bitswap asks?) for too many CIDs at the same time, and were considering batching

What max batch size would make sense for reframe?

[ajnavarro]

What max batch size would make sense for reframe?

It depends on what we think is an acceptable and sane default for an HTTP payload size. 10Mb? 100Mb? 2Gb?

Here are the POST body sizes for some of the more popular HTTP servers:

• Nginx (largest web server market share as of April 2019) - default 1MB, no practical maximum (2**63)
• Apache - maximum 2GB, no default documented
• IIS - default 28.6MB for the request length, 2048 bytes for the query string; maximum undocumented
• InfluxDB - default ~25MB, maximum undocumented

[lidel]

Given the IPNS record size being capped at ~10KiB, and block limit in IPFS-over-libp2p being ~1MB atm, I'd go with 1MB or something in-between.

[BigLep]

Per @ajnavarro discussion in this issue and in 2022-11-07 standup, please also include a spec change change with a recommendation on limits.