Torrent does not download when removing read block from cache

[cocool97]

I'm trying to implement a custom TorrentStorage where pieces would be remove from this storage when being read by a client, to prevent much memory from being used.

The implementation is quite simple, and does not change much from your custom InMemoryStorageExample implementation.

When I keep all pieces in storage, everything works fine, but obviously memory goes up. With the following code, torrent does not download at all :

fn pread_exact(&self, file_id: usize, offset: u64, buf: &mut [u8]) -> anyhow::Result<()> {
        log::debug!("pread_exact {file_id} {offset}");
        let fi = &self.file_infos[file_id];
        let abs_offset = fi.offset_in_torrent + offset;
        let piece_id: u32 = (abs_offset / self.lengths.default_piece_length() as u64).try_into()?;
        let piece_offset: usize =
            (abs_offset % self.lengths.default_piece_length() as u64).try_into()?;
        let piece_id = self.lengths.validate_piece_index(piece_id).context("bug")?;

        let mut g = self.map.write();
        // Get and remove this data from buffer to free space
        let inmp = g.remove(&piece_id).context("piece expired")?;
        buf.copy_from_slice(&inmp[piece_offset..(piece_offset + buf.len())]);

        Ok(())
    }

Are you for example reading the Storage when a block has been added using pwrite_all ? This would explain the issue I have as block would be remove directly after being downloaded.

I'm using rqbit vendored from git directly, with latest version v7.0.0-beta3

[ikatson]

Are you for example reading the Storage when a block has been added using pwrite_all ? This would explain the issue I have as block would be remove directly after being downloaded.

Of course. After the peer sends all the piece chunks, they are re-read to get hashed. So you can only remove it after that technically. You also need to disable upload, disable Have messages sent to peers as it'll fail reading.

[cocool97]

How can it be done programmatically ?
I don't think that I have control over it in the TorrentStorage trait, and the goal is to not have everything in RAM at the end of the download

[ikatson]

I'll be fine with a PR to add smth like "on_piece_completed" callback to TorrentStorage

[cocool97]

Ok seems good yes, I'll handle it ;)