Downloader for iroh-bytes

[Frando]

The iroh-sync PR adds a downloader. In its current state it is unfinished, and cannot do retries and doesn't have a proper queue.

Features we need from the downloader from a high level iroh sync point of view:

• Enqueue downloads as Hash, PeerId pairs
• Get notified when a download is finished (through a future)

This high level API exists already in the PR, with public methods push(hash, peer_id) and async finished(hash: Hash). The actual actor below misses many features though. We want:

• Retries with incremental backoff
• Limits on parallel downloads and connection attempts (e.g. max 5 concurrent dials)
• Being able to queue a hash with a list of peers in an order with different timeouts (e.g. try from peer 1 immediately, from peer 2 in 100ms, and from peer 3 in 1 second) - aborting/skipping of course if a download succeeded

--

copying some notes from the now-merged downloader improvements PR:

The Downloader currently acts immediately on new requests and doesn't do any retries. It also does not have a proper queue internally.

We want to be able to queue downloads with retries and different timeouts. My idea was to add a time-based queue, maybe using the TimerMap from iroh_gossip::util. Then I thought to change the API to expose download(hashes: Vec<Hash>, peer: Peer, delay: Option<Instant>). The actor would maintain a timer queue of pending requests and their delay, and poll that for the next operation, check if the operation is still needed, and if so enqueue the future to actually do the download. The actor should also keep track of running downloads, and have a limit of how many peers are being dialed or connected to in parallel (configurable). Code-wise, I thought to add at least a DownloadState for each pending download or so, and then similar to the current code have indexes into those for PeerId and Hash (and Instant if using something like the TimerMap).

With this, we can then:

• On new entry message via gossip, enqueue download from
  • peer we recv'd from with a very small delay
  • from our other current active gossip neighbors with a larger delay
  • from the author with a larger delay plus random offset (*note: for this we need to add the peer info (address&region), with a signature, to the gossip message in addition to the SignedEntry)
• Queue more retries if all fail up to some limit

Later on, we'd also want to add an explicit command to retry all missing hashes in a document from a peer, and optionally a mode that would do that automatically if we get new neighbors. This will initially be quite inefficient for large docs, because we have no ẁant/have` exchange in iroh-bytes. So maybe defer this a bit.

Edit
Another thought on the API, with peer and delay we'd also want to pass a enum RetryStrategy { None, Auto } or so likely, with auto being some incremental backoff with max retries or similar.

Edit 2
Currently the quinn::Connections are dropped and thus cancelled if there is no immediate next download pending after a download finished. Likely we'll want to keep them around for a configurable-with-default amount of time, and then drop them if no operation happened. Redialing is quite cheap but not free (one roundtrip in current setup).

[rklaehn]

A few thoughts on this topic:

I think the downloader API should be separated into declarations of intent and hints.

A declaration of intent would be e.g.

• I want to have <hash>.
• I want to have ranges <ranges> of <hash>
• I want to have collection <hash> and it's children

For a declaration of intent you get a u64 handle that allows you to control/delete the intent.

Then you have hints. Hints are something like:

• you might be able to get <hash> from <peer>, says <source>, or some more complex hint

It is then up to the downloader to make use of the hints, ask some discovery mechanism etc.

The downloader keeps track of the intents and makes sure that there are no duplicates, e.g. if two places want the same hash we don't want two download tasks.

An extension of the intent/hint scheme would be something like topics. E.g. instead of saying for these 10000 hashes you should try these 100 peers, have a "download topic" that in case of doc sync might correspond to a document. Then you can say "I want to download these 10000 hashes for topic <x>", and "these 100 peers are participating in topic <x>".

A big question: should the downloader support collections? I would say yes, otherwise the downloader will be useless for many non document sync related iroh tasks.

[divagant-martian]

The parts that @Frando mentions seem clear to me how to implement, @rklaehn 's not so much.

Could you guys help me here? (sorry still noob to these parts of the code):

• Given a hash that is collection, how do I retrieve the hashes that belong to the collection? Last time I checked the collection decoding has a list (set?) of hashes. Please correct me if I'm wrong or if this has changed
• Given a hash, how to map a range to a list of hashes?

Thank you both!

[Frando]

Given a hash that is collection, how do I retrieve the hashes that belong to the collection? Last time I checked the collection decoding has a list (set?) of hashes. Please correct me if I'm wrong or if this has changed

You need the CollectionParser that is part of an iroh node. That likely means that the Downloader will have to be generic over C: CollectionParser, and the collection parser will have to be passed into the downloader. You can then call collection_parser.parse() to obtain a list of hashes from a collection. The CollectionParser can be set by the application to support other kinds of collection than the default iroh collections.

Given a hash, how to map a range to a list of hashes?

Not sure I got the question right, and @rklaehn knows much more than me about hashes and ranges. I think a range always refers to part(s) of a single blob identified by a single hash. So you can say "gimme these ranges of this blob identified by hash". I do not know how this relates to collections and if we have a concept of "ranges of a collection".

[rklaehn]

The parts that @Frando mentions seem clear to me how to implement, @rklaehn 's not so much.

Could you guys help me here? (sorry still noob to these parts of the code):

• Given a hash that is collection, how do I retrieve the hashes that belong to the collection? Last time I checked the collection decoding has a list (set?) of hashes. Please correct me if I'm wrong or if this has changed

What @Frando said

• Given a hash, how to map a range to a list of hashes?

The protocol for individual blobs goes like this: you ask for a hash and a set of ranges of chunks (1 chunk = 1024 bytes). Then you get back data only for what you have asked for. The default thing is to just ask for a hash and all it's chunks RangeSet2::all(), then you get all the data for the blob.

For collections it is more complex. You can specify ranges for the collection itself and for each of it's children. E.g. you have downloaded a collection and half of it's children and were interrupted. You then figure out what is missing and ask for exactly that.

Thank you both!

[rklaehn]

Here is an issue that is related to the downloader. Probably not initially, but eventually we want to support this: #839

[b5]

@Frando & @divagant-martian both agree that #1420 closes