Support WebTorrent Protocol

[kevincox]

WebTorrent is a new implementation of the bittorrent protocol designed to work over WebRTC. This allows browsers to download torrents without plugins. However, due to the limitations of WebRTC it isn't possible to make the protocol compatible with bittorrent (or uTP).

If libtorrent supported WebTorrent it would be able to communicate with both the existing swarm as well as the browser swarm that is restricted to WebTorrent only.

[DiegoRBaquero]

+1

[B00tLdr]

+1! This would make it possible to have something like popcorntime natively in the browser :D
Maybe use something like: https://github.com/xhs/librtcdc for this. @feross could maybe add some info if this is possible/easy.

[RobinHoutevelts]

Having support for WebTorrent would be huge

[kevincox]

I would also like to reiterate that while having a good swarm of bittorrent, and a good swarm of webtorrent are both really important, libtorrent would be in the critical situation of bridging the two which is critical to the success of webtorrent.

[ghost]

Huge +1, I think Webtorrent integration is part of the future of Bittorrent; casual swarms and torrent streaming in-browser are killer apps.

edit: I always mix up the names for Bitcoin and Bittorrent, sorry!

[ngosang]

+1

[gubatron]

WebTorrent should be the one to submit a patch with the WebRTC transport implementation.

AFAIK they had to change a few things on their side bittorrent-protocol wise to make it work on browsers, who better than them to do this.

As of now, there's simply two separate networks, without libtorrent having this transport the WebRTC will never be able to access the seeds on the TCP/UDP side.

However, from a practical standpoint, will people keep a browser tab open to seed? possibly creating memleaks on their browser, slowing down their day to day work? or will this just add a massive number of seeds that will ultimately be a huge performance hit for the existing bittorrent network?

[kevincox]

However, from a practical standpoint, will people keep a browser tab open to seed?

This is hard to say, however with web technologies quickly growing there is no reason why some people wont run their torrents in the background, just like the do with native apps (no tab required). I feel the line between desktop and web is growing thin and not providing features to web apps because they are web apps is becoming an antiquated view point. You may be right, but I suspect that the playing field will change very quickly.

[transitive-bullshit]

@arvidn I'm a webtorrent contributor interested in adding support to libtorrent. Do you have any advice on where to start or potential gotchas to be aware of at the outset?

[aldenml]

@fisch0920 My suggestion is start digging peer_connection and peer_connection::start for clues of how to create a possible "webrtc_peer_connection". Do you have a WebRTC library candidate, or you want to recreate the protocol?

[yipperr]

@kevincox i agree the line between apps and browsers are going thin there is also the issue where webtorrent might negatively impact the swarm by allowing sequential file transfer of media files to be selected the relative ease for media files to be streamed into the video tag of the browser might make it appealing for people to opt for sequential downloads more often that the rarest first and also since lot of people noticed the synergies and obvious usage as this popcorn time in a browser this kind of compounds the facts and might really cause concern on swarm stability

[arvidn]

@fisch0920 aldenml's advice is right btw. the peer_connection is the base class connecting peers to the rest of the bittorrent engine. bt_peer_connection derives from it and implements the bittorrent protocol, there's also web_peer_connection and http_seed_connection for the two kinds of web seeding protocols. deriving from peer_connection to implement a new peer protocol is how it's intended to work. now, I wouldn't be surprised if there will be some rough edges, as its interface has been designed around bt_peer_connection

[obv-mikhail]

Might be possible to use this https://github.com/mozilla/webrtc-standalone for webtorrent support.

[aldenml]

Do you see this in the Makefile?

LIBS = \
$(GECKO_OBJ)/mfbt/libmfbt.a.desc \
$(GECKO_OBJ)/layout/media/webrtc/libwebrtc.a.desc \
$(GECKO_OBJ)/media/libopus/libmedia_libopus.a.desc \
$(GECKO_OBJ)/media/libvpx/libmedia_libvpx.a.desc \
$(GECKO_OBJ)/media/libjpeg/libmedia_libjpeg.a.desc \
$(GECKO_OBJ)/media/libspeex_resampler/src/libspeex.a.desc \
$(GECKO_OBJ)/netwerk/srtp/src/libnksrtp_s.a.desc \
$(GECKO_OBJ)/media/mtransport/standalone/libmtransport_standalone.a.desc \
$(GECKO_OBJ)/media/webrtc/signalingstandalone/signaling_ecc/libecc.a.desc \
$(GECKO_OBJ)/xpcom/libxpcomrt/libxpcomrt.a.desc \
$(GECKO_OBJ)/dom/media/standalone/libmedia_standalone.a.desc \
$(GECKO_OBJ)/netwerk/standalone/libnecko_standalone.a.desc \
$(GECKO_OBJ)/intl/unicharutil/util/standalone/libunicharutil_standalone.a.desc \
$(GECKO_OBJ)/security/nss/lib/pk11wrap/static/libpk11wrap_s.a.desc \
$(GECKO_OBJ)/security/nss/lib/freebl/static/libfreebl_s.a.desc \
$(GECKO_OBJ)/db/sqlite3/src/libdb_sqlite3_src.a.desc \
$(GECKO_OBJ)/memory/fallible/libfallible.a.desc \
$(GECKO_OBJ)/media/libyuv/libyuv_libyuv/libyuv.a.desc

how many MB is this standalone library?

I think a good starting point is find a minimal WebRTC communication stack, or figure out a clear interface to plug inside libtorrent as another protocol.

[jscinoz]

@aldenml https://github.com/xhs/librtcdc suggested by @B00tLdr should be a good fit - it's certainly much more lightweight than webrtc-standalone, since librtcdc it implements data channels only, whereas webrtc-standalone implements all the media related functionality of WebRTC too, which isn't necessary for WebTorrent support.

[aldenml]

yes, I saw the link right after I left my comment, thanks

[arvidn]

Just for the record. The receive buffer in peer_connection right now is a complete mess. Mostly because it provides "convenience" functions to advance read pointers and store the current message size being received. Things that are mostly useful for the bittorrent peer, but not really for anyone else. So, if anyone will be looking at integration with webRTC, it would probably make sense to simplify the receive buffer too.

[dessalines]

I'd like to help out with webRTC support, where would be a good place to start?

EDIT: Yo @aldenml , I'd like to start working on this. I have libtorrent building correctly on my machine, but I suppose I need a good way to test out changes that I've made, IE, how to run the client_test.

Also, I'm familiarizing myself with peer_connection.cpp, but I'd definitely need some help figuring out the architecture, mainly where the best place to put the new connection type should be, and how best to include the suggested webRTC library, https://github.com/xhs/librtcdc .

[dessalines]

@aldenml Sorry if this pinged you twice, I edited my comment above to ping you, but I'm not sure if github edits will re-ping someone. If there's some other client like IRC or gitter that would be better for possibly bringing me up to speed on this, let me know.

[aldenml]

not a problem (and thanks). I don't have experience with any example in libtorrent, only the tests.

[adamvoss]

@dessalines Thanks working on this! I have dug into your questions.

Running client_test

client_test is included in the repo under the /examples folder. Executing make client_test from within that directory compiles it and puts the result in the same directory; alternatively, there is an entry in 'examples/Jamfile' if your are using bjam rather than make. Then a simple ./client_test <filename.torrent> in that directory to run it.

Where to get started

arvin has advice in this comment. Paraphrasing:

peer_connection is the base. There are 3 implementations: bt_peer_connection, http_seed_connection, and web_peer_connection

You would start with a new implementation, webtorrent_peer_connection (or similar), deriving from peer_connection

by convention, it would appear your ".hpp" file would go in /include/libtorrent and your ".cpp" file in /src

Visual representation of hierarchy:

peer_connection
└─ bt_peer_connection
└─ web_connection_base
│  └─ http_seed_connection
│  └─ web_peer_connection
└─ webtorrent_peer_connection    ( <-- you add this)

How to include librtcdc

I am not familiar enough with the repository to say definitively. However, you can always get started and move the files later. For the sake of development, what I would do is just use git submodule to pull down the code at the root of the project.

git submodule add https://github.com/xhs/librtcdc.git

(I am guessing the root because I see ed25519 has its own folder at the root). Later it can be decided whether a submodule is okay or if a copy of the source will be included and where the final location should be.

[arvidn]

I think pulling in librtcdc as a submodule would probably be best. one challenge I would expect is that the peer_connection class has a pretty rigid "framework". It owns the socket, it implements rate limits, SOCKs proxies, some send- and receive- buffer management and some other high level features. The receive buffer part is not great unfortunately, it's fairly tied to BitTorrent protocol. You specify how many bytes you expect to receive and then you get called every time more bytes are received.

If librtcdc itself owns sockets, that might make it difficult (at least if you want to integrate with the main message pump/asio.io_service in libtorrent). It the library let you pass byte in and it passes bytes back out again, it should make it easier to integrate on top of libtorrent peer connections.

About the headers, I'm hoping to move towards having only public headers live in include/libtorrent and have private headers live in include/libtorrent/aux_ (and in the libtorrent::aux namespace). Some internal classes do that, but not all yet.

[ducdigital]

+1 any progress so far?

[nkosi23]

@gubatron @yipperr

or will this just add a massive number of seeds that will ultimately be a huge performance hit for the existing bittorrent network?

i agree the line between apps and browsers are going thin there is also the issue where webtorrent might negatively impact the swarm by allowing sequential file transfer of media files to be selected the relative ease for media files to be streamed into the video tag of the browser might make it appealing for people to opt for sequential download

I read the bittorent white paper a long time ago so my knowledge of the protocol is not very deep, so you really triggered my curiosity here. Why would a large number of seeds affect the performance of the network ? And what's the deal with sequential file transfers ? Could you please introduce me to the situation, or give me links me to relevant documentation (whichever is the easiest for you).

Many thanks !

[bobpoekert]

There will be two implementations:

1. An unsecured websocket only p2p implementation with trackers and every other protocol (DHT, PEX).
2. A websocket signaling WebRTC p2p implementation with trackers. This is limited because certificate authority crap. I'm upset.

Dealing with SSL requirements and certificate authorities makes a free form swarm impossible. See you in a month or three if no one takes care of this before then. I need to go into my cave.

Even if it weren't for ssl websocket is based on tcp so you can't do hole punching

[backkem]

There will be two implementations:

1. An unsecured websocket only p2p implementation with trackers and every other protocol (DHT, PEX).
2. A websocket signaling WebRTC p2p implementation with trackers. This is limited because certificate authority crap. I'm upset.

Dealing with SSL requirements and certificate authorities makes a free form swarm impossible. See you in a month or three if no one takes care of this before then. I need to go into my cave.

Even if it weren't for ssl websocket is based on tcp so you can't do hole punching

The websocket transport is only used to connect to trackers. There you don't need hole punching. For the p2p connection WebRTC takes care of hole punching.

[paullouisageneau]

I recently wrote a light library implementing WebRTC Data Channels in C++ to power another of my projects: https://github.com/paullouisageneau/libdatachannel
It is still young but it should be fully functional already (more examples and tests are coming). It's under LGPLv2, and I'd be glad if it can be useful to someone else.

[paullouisageneau]

I opened a draft PR with a WebTorrent implementation based on the libdatachannel library: #4123
It works, but still needs testing!

[transitive-bullshit]

This is absolutely amazing work @paullouisageneau. Really excited to see where this work leads! 😄

[shirishag75]

This is absolutely amazing work @paullouisageneau. Really excited to see where this work leads! smile

The great part about this is that @arvidn is actually testing and asking him questions on the draft PR and @paullouisageneau is making the changes based on the queries or suggestions made by arvidn . This means there is a high possibility that after modifications and any cleaning up required, it may get integrated in libtorrent in a future release which furthers clients such as qbittorrent and other apps. which use libtorrent. I, for one am looking for this update, as well as I'm sure others are.

[paullouisageneau]

Thanks @transitive-bullshit !
There is still work to be done on it for sure, since the modifications are quite extensive 😅

[shirishag75]

Thanks @transitive-bullshit !
There is still work to be done on it for sure, since the modifications are quite extensive sweat_smile

@paullouisageneau I only hope you keep us in the loop. Would all the modifications be back-ported upstream or your or arvidn end up carrying patches ? How do you see it happening ?

Edit - I do see that @arvidn seems to have raised lot of queries and provided hints of how some things could be better in src/rtc_stream.cpp . Hope we get it all worked out and imbubed in libtorrent. I did notice arvidn talking about C++17 change and hopefully we see that happening after libtorrent 12 comes out.

[backkem]

FYI, There is a new C WebRTC implementation from AWS: https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-c/tree/master/src/source
Fun fact: It happens to be built by the creator of pion/webrtc, the Go WebRTC implementation.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[filips123]

It's still work in progress (#4123) so I don't think it should already be closed.

[disarticulate]

It's still work in progress (#4123) so I don't think it should already be closed.
Agreed.

[arvidn]

I thought I had configured the bot to leave PRs alone, but I guess not oh, right. This is the ticket

[FranciscoPombal]

@arvidn you can create a new label named "no-stale", or "important" or something like that, and add it to stale bot's exempt labels.

[Mikaela]

Judging by the current stale.yml, this should be labeled as pinned to not get staled, alternatively security but it seems a bit inappropiate to me.

libtorrent/.github/stale.yml

Lines 6 to 8 in 9f3192f

	exemptLabels:
	- pinned
	- security

[mmathys]

Is this WIP?

[paullouisageneau]

Yes, this is still WIP. We're getting there!

[paullouisageneau]

Implementation-wise it's finished, it could be merged after reviews and bump to C++17.

[paullouisageneau]

I think this can be closed now #4123 is merged!