WebTorrent should not store torrent data in memory

[feross]

Description

WebTorrent seems to be keeping the entire torrent in memory as the file is downloaded. I can see the RAM usage of the "Brave Browser Helper" process increase along with the amount of the torrent that is downloaded, which should not happen. It appears that it is being initialized with the in-memory storage mode instead of the filesystem mode. This is because webpack is used to build the JS for the WebTorrent extension and that causes WebTorrent to substitute certain packages using the package.json "browser" field.

May be related to: #5372

Steps to Reproduce

1. Visit http://mirror.math.princeton.edu/pub/mageia/iso/7.1/torrents/Mageia-7.1-x86_64.torrent
2. Start the torrent download
3. Open Activity Monitor.app on Mac and observe memory usage climbing in proportion to the amount of torrent that is downloaded.

Actual result:

Torrent is stored in memory instead of on the disk.

Expected result:

Torrent should not consume RAM in proportion to the torrent size, since this means that users with e.g. 8GB RAM are going to have a sad time when they download an 8GB+ torrent.

Reproduces how often:

Easily reproduced

Brave version (brave://version info)

All Chromium versions.

[btlechowski]

Similar/Dupes issues #1199 #1228

[sheddup]

Is this being looked at? Can something like https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API be used instead of RAM

[ftoh]

Is this being looked at? Can something like https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API be used instead of RAM

I would prefer it should store files in Downloads folder to be able open with other players (for media).

[tudor-pop]

I looked into the same thing these days. I believe the answer you are looking for is to use an Abstract Chunk Store implementation. Unfortunately idb-chunk-store has some issues when I tried to use it last day. The one that worked for me(but not perfect) is this one. The thing is that both of these were not updated since last 3+ years ago so I don't know if it could be improved. My usecase is to seed large files(5GB for example). Using memory-chunk-store it crushes my RAM and occupying 8GB in memory data. With indexeddb-chunk-store goes to like 5GB but it also takes a bit more time to finish the seed(since it writes to disk). For browser I'm not sure anything can be done because of lack to user FS.
I also tried the webtorrent-cli (which has access to the FS) and is almost instant when seeding a file with no memory footprint(that I could see in activity monitor) so my guess is that is using the FS-chunk-store. I'm quite new to this project(and browser/javascript tech in general) & some confirmation from someone in the official team would be great. I know the thread is for Download but I think is the same issue for seeding

[tudor-pop]

update: idb-chunk-store seems to be fine after removing package-lock.json and npm install again. However the seeding of a 5GB file still takes 5GB of RAM

[MatheusPese]

This issue is bad because my pc just freezes completelly when downloading torrents with brave because of it. My RAM usage goes to 100%

[srirambv]

This should probably be prioritized and fixed soon. Torrent feature becomes useless if the browser starts to freeze up and download is lost.

cc: @rebron @bsclifton @feross

[AerosHawk]

Can confirm this is still an ongoing issue.
I was also trying to find out if using webtorrent inside Brave protects you from sneaky ISPs, the same way that a VPN does..? Sorry, side point, but wasn't really able to find an answer and it led me here.

[loic-bellinger]

Any update on this? Very disappointing.

[srirambv]

+1 from @daphnei-mee via #14814

[feross]

Hi everyone,

Thanks for all the feedback. I have a plan for fixing this issue and reducing RAM usage.

We'll use a chunk store that stores data to disk instead of to memory. Many of the existing chunk store options available for use in the browser have disappointing performance, which is why I haven't wanted to make the change yet. The IndexedDB chunk stores in particular have historically had atrocious performance.

I've done a thorough investigation of the existing stores and discovered many low-hanging performance improvements in the chunk stores, but also in the core WebTorrent library itself. It's now looking like we can reasonably use one of these and still get good performance.

Finally, I've been developing a new chunk store based on the new Filesystem Access API which has shipped in Chromium-based browsers. It performs almost as good as the memory-chunk-store (the current solution) but, of course, uses no RAM.

The only remaining issue to figure out is how to handle clearing the data. We'll need to make some decisions about when is the right time to do that – when the user closes the tab? when they stop the torrent? when they explicitly click "delete"?

[feross]

Regarding data deletion, here's a proposal:

I assume it's fine to write data to disk using browser storage APIs, at the user's request. It's no different than what any random website can already do.

If we use a browser storage mechanism like the Filesystem Access API, then data will go into the same place as the rest of browser storage and be cleared automatically by the same eviction policies. We can set the Filesystem Access API to temporary mode so that data will automatically be cleared by the browser when disk space is low. Futhermore, when the user stops downloading the torrent, we can automatically clear the disk space.

Beyond that, is there anything else I need to think about?

[bsclifton]

It makes sense (to me) to treat content downloaded via WebTorrent the same as when you download a file from a website

Given that, I don't know if it would ever go out of scope unless you are in a private / Tor window and close all the tabs. Even then, I think Chromium/Brave will keep downloads on disk, it just won't be listed in the download manager interface

[brasscupcakes]

Not sure why Brave doesn't just do away with webtorrent as a default add-on at this point.
Nobody actually prefers using it to using a standalone torrent client and it doesn't generate any revenue.
Years in, most users still consider it an annoyance -- why waste developers' time "fixing" a feature nobody wants when so many valuable features beg for attention?

[jjbofficial]

Not sure why Brave doesn't just do away with webtorrent as a default add-on at this point.
Nobody actually prefers using it to using a standalone torrent client and it doesn't generate any revenue.
Years in, most users still consider it an annoyance -- why waste developers' time "fixing" a feature nobody wants when so many valuable features beg for attention?

I disagree. I think that as someone who uses torrents frequently just like a lot of people in this community, this feature is valuable to us.

[polarathene]

why waste developers' time "fixing" a feature nobody wants when so many valuable features beg for attention?

Nobody wants it perhaps due to the memory issue discussed in this thread? Fixing that by supporting disk downloads instead would change that.

---

My interest is from the opposite side of a user, a community website with large downloads (game mods) where we can offload bandwidth expenses through torrents. Some users may not be comfortable with third-party torrent software and it's easier to support them via webtorrents and advising a client like Brave making it seamless in the users browsing experience.

For other users comfortable with a separate desktop app, they can use that along with magnet links. I know it's not a big ask for that, but some users do consider that as enough friction vs their comfort of using a web browser only to perform a download.

We are not too big, but reducing the 20TB we average monthly would be good, especially when a download gets very popular (last month one download link alone used 8TB). Torrents can really help there, and the easier it is for transitioning our users to the better.

[polarathene]

Futhermore, when the user stops downloading the torrent, we can automatically clear the disk space.

This will only make seeding possible during the download/leeching?

I know it's common for leechers to stop seeding not long after they complete the transfer, but could Brave also provide opt-in seeding for some duration or seed ratio? If not, that's fine :)

[plehman2000]

Is this ever getting fixed? This feature is literally unusable if the torrent's size exceeds your RAM

[sandesnp]

Problem persists to this day. I don't understand why download via webtorrent isn't treated same as a regular download aka filesystem.

[tudor-pop]

Problem persists to this day. I don't understand why download via webtorrent isn't treated same as a regular download aka filesystem.

we don't have access to the OS in the browser because of sandboxing...we need to use Filesystem Access API which is broadly barely supported

[plehman2000]

Since it's open source, does that mean one of us is eventually supposed to fix it?

[tudor-pop]

@plehman2000 I guess so...I looked over it for almost a week trying to figure it out but then I decided to use plain WebRTC...Also the issue is not specific to the Brave browser..is a browser general issue...

[3vilcrow]

I am using webtorrent from the server and it consumes the same ram as the file to download, I have used the got library to download by stream but it had never given me this ram memory expense. Another alternative?

[nisnevich]

Sometimes I forget to save the file after downloading, and it's so frustrating. I found this issue while looking for a way to retrieve file from cache instead of downloading again...

[alexruimy]

Webtorrent is one of the worst integrations of anything I have ever seen.

[neutralme]

The problem still exists. Is anyone looking over this issue? The time when page says "Aww snap......" I literally want to blow my PC.

[fallaciousreasoning]

Hopefully, in the next few weeks I might be able to take a look, no promise though. This is on my todo list 😄

[Nyanpasu1]

i will keep it disabled until it is fixed, i will use qbitorrent for now

[fallaciousreasoning]

@feross, I've been playing around with using this package instead of the in-memory-chunk-store. Similar to what @tudor-pop found, the memory usage is marginally better, but not as much as I'd expect (still requires approximately the same amount of ram as the file size). I also tried turning off it's internal chunks cache, which mildly improved memory usage.

Performance is much better than the idb-chunk-store though.

Note: Even though this package is called fs-access-chunk-store it's just using the old FileSystemFileHandle APIs, rather than the new high performance ones.

[Animatron1]

It has been almost 4 years now, and absolutely nothing has been done about this since then. I am forced to use Transmission whenever I download anything over 10GB. Why is this feature still here if it's neglected and flawed.

[Xarius86]

It's been five years now, and this computer crashing bug is still being shipped.

[cashlycash]

Wow. So is this issue being completely ignored?

[Animatron1]

This is beyond ridiculous at this point. 5 years have gone by without the developers paying any attention to a problem that renders the webtorrent feature useless for most users. It also seems like an easy problem to fix, requiring the extension to save torrent downloads directly to disk instead of RAM.

Please have mercy on us, give us a glimmer of hope. How can you brand yourself as a "Web 3.0 browser" if you neglect one of the most important decentralization features - the torrents?

@fallaciousreasoning, @feross, @rebron, @bsclifton