rTorrent Performance Tuning - proposal for a modern .rtorrent.rc

[bbsixzz]

As previously discussed in #3273 there were many deprecated commands in what could only be described as a years-old rotting piece of garbage. Even after corrections there is much left to be desired; clarity, performance etc…

I've since done my homework and would like to share a new configuration that prunes everything superflous while also increasing performance according to rTorrent Performance Tuning.

I've been testing it for a couple of weeks using two RPi4's (4GB) on two separate networks (both behind WireGuard VPN) with great results in stability and seeding/leeching performance.

# Connection settings
network.port_open.set = no
network.port_random.set = no
network.scgi.open_port = 127.0.0.1:5000

# Network limits
network.http.max_open.set = 50
network.max_open_files.set = 600
network.max_open_sockets.set = 100

# Peer settings
throttle.min_peers.normal.set = 39
throttle.max_peers.normal.set = 40
throttle.min_peers.seed.set = -1
throttle.max_peers.seed.set = -1
throttle.max_downloads.global.set = 100
throttle.max_uploads.global.set = 100
throttle.max_downloads.set = 20
throttle.max_uploads.set = 20
trackers.numwant.set = 40
trackers.use_udp.set = no
dht.mode.set = disable
protocol.pex.set = no
protocol.encryption.set = allow_incoming,try_outgoing,enable_retry

# Miscellaneous settings
system.umask.set = 002
system.daemon.set = true
pieces.memory.max.set = 1000M
pieces.hash.on_completion.set = no
directory.default.set = /mnt/server
session.path.set = /mnt/dietpi_userdata/downloads/.session
schedule2 = monitor_diskspace, 15, 60, ((close_low_diskspace, 1000M))
schedule2 = session_save, 240, 300, ((session.save))

network.port_open.set = no
Default 6881-6999, disable if using VPN (which you should).

network.port_random.set = no
Disable for the same reason as above.

network.scgi.open_port = 127.0.0.1:5000
This needs to be set in order for ruTorrent to work.

network.http.max_open.set = 50
Maximum number of simultaneous HTTP request (used by announce/scrape requests).

network.max_open_files.set = 600
Maximum number of open files rtorrent can keep open.

network.max_open_sockets.set = 100
Maximum number of connections rtorrent can accept/make (sockets).

throttle.min_peers.normal.set = 39
throttle.max_peers.normal.set = 40
Maximum and minimum number of peers to connect to per torrent while downloading.
We always want to ask the tracker for new peers more frequently, using tracker min interval.

throttle.min_peers.seed.set = -1
throttle.max_peers.seed.set = -1
Same as above but for seeding completed torrents, -1 for same value.

The min_peers values are responsible for asking more peers during an announce request. When the client has less than min_peers - (peer_list_size / 2) connections for a download and PEX doesn't result in connection for a download (if it's enabled at all), it will attempt to request more from available trackers using tracker min interval, otherwise using tracker interval. 30 seconds after a request the client will attempt another if more than 10 new peer connections were gained or less than 3 requests have been performed. Else it will try the next tracker group in the list, but not other trackers in the same group. This behavior should give enough peers while minimizing the number of tracker requests, although it will use somewhat longer time than other more aggressive clients.

throttle.max_downloads.global.set = 100
throttle.max_uploads.global.set = 100
Maximum number of simultaneous downloads and uploads slots (global slots!).

throttle.max_downloads.set = 20
throttle.max_uploads.set = 20

Maximum number of simultaneous downloads and uploads slots per torrent
.

trackers.numwant.set = 40
Set the numwant field sent to the tracker, which indicates how many peers we want.

trackers.use_udp.set = no
Following template instructions.

dht.mode.set = disable
protocol.pex.set = no
Conservative settings for private trackers.

protocol.encryption.set = allow_incoming,try_outgoing,enable_retry
Following template instructions.

system.umask.set = 002
This is untouched from old config, new default is 0027.
@MichaIng maybe you care to comment?

system.daemon.set = true
Run as a daemon, no need to change unless for testing purposes.

pieces.memory.max.set = 1000M
Commonly misunderstood setting: it doesn't limit or set the amount of RAM rTorrent can use but limits the amount of memory address space used to mapping file chunks. This refers to memory mapping, not physical memory allocation. Default value is 1 GiB.

pieces.hash.on_completion.set = no
Disabled, intensive processing to check file integrity after a finished download.

directory.default.set = /mnt/server
Location of your downloads.

session.path.set = /mnt/dietpi_userdata/downloads/.session
Location of your session files.

schedule2 = monitor_diskspace, 15, 60, ((close_low_diskspace, 1000M))
Check for low diskspace.

schedule2 = session_save, 240, 300, ((session.save))
Save sessions every 5 min, you may lose resume/statistical data if you terminate prior to a save.

[MichaIng]

@bbsixzz
Many thanks for this, an overdue enhancement indeed. I'll go through it in detail but on first view it looks very good 👍.

[bbsixzz]

I'm glad I could help, thanks!

I forgot to explain the following relationship:

throttle.max_peers.normal.set = 40 
throttle.max_peers.seed.set = -1

throttle.max_downloads.set = 20
throttle.max_uploads.set = 20

max_peers settings for downloading and seeding should be at least 2 times higher than the number of slots per torrent according to rTorrent Performance Tuning.

This in turn works out to 5 concurrent downloads and uploads saturating the global settings:

throttle.max_downloads.global.set = 100
throttle.max_uploads.global.set = 100

It looks nice too compared to the old configuration that left a lot of odd values and/or blanks.

[MichaIng]

@bbsixzz

max_peers settings for downloading and seeding should be at least 2 times higher than the number of slots per torrent

Jep, I read that as well. Funny is that their own default config template does not respect that rule 😄. However very reasonable according to what those two settings mean.

[bbsixzz]

I would like to make an addition to the config after running into this issue rakshasa/rtorrent#731

# Connection settings
network.local_address.set = 127.0.0.1

The client now downloads correctly.

Fixed in v0.13.8 rakshasa/libtorrent@0ee9799 but until then should be kept in config.

[EDIT]

After reboot I started getting Invalid IP detected from the previously unaffected trackers.

I think I need to try this solution and report back before I do anything further.

[EDIT]

I decided not to mess with dnsutils, the only tracker affected by this is EMP :]

network.local_address.set = [Your WireGuard endpoint IP] works fine if you really need it.

[MichaIng]

@bbsixzz
Okay lets go through the settings: https://github.com/MichaIng/DietPi/blob/dev/dietpi/dietpi-software#L10595-L10715

• network.port_open.set basically true, this has no effect anyway unless user is not explicitly open/forward the port in router and most VPN provider do not support port forwarding, hence there is has no effect as well. Seeding still works when rTorrent searches/connects to peers via outgoing connections. So it makes sense to keep all of this disabled by default and leave action to user to open the listening port it wants. Btw I guess network.port_random.set as no effect if port_open is set to off anyway? So could be left commented together with port_range setting, so users can quickly identify/enable those two that belong together.
• network.scgi.open_port: What do you think about using a local socket only by default? network.scgi.open_local = /mnt/dietpi_userdata/downloads/.session/rpc.socket + execute.nothrow = chmod,777,/mnt/dietpi_userdata/downloads/.session/rpc.socket. But this would need the webserver configs to be changed as well... postponed to a later release.
• network.max_open_files.set we use 50 for this currently. I think 600 is way to much for a SBC? Probably we should use max_downloads.global + max_upload.global here?
• throttle.max_downloads.set is estimated based on arch/SBC currently, 30 for a Gbit Ethernet SBC. throttle.max_peers.normal.set we use on Gbit network SBCs (( 30 / 2 + 1 )) currently, however indeed it makes more sense to swap this, so that not less peers are allowed to connect to a torrent then downloads can be done. I will keep our per-arch/SBC function in use but swap the settings according to your example and official performance recommendations.
• pieces.hash.on_completion.set = no yes this takes time, but I would prefer to keep this on to be sure that a download went successful in the first place instead of running into (playback or whatever) failure later.
• What do you think about keeping the ration settings in place or not? Sounds reasonable to me to close a torrent automatically when seeding ratio is reached? But if that is more something that one wants to define individually, then probably the defaults are okay, whatever they are.
• protocol.encryption.set leave it required, for security/privacy reasons?
• system.umask.set = 002 yes this is required, in relation with dietpi group membership, to assure that other programs like Sonarr/Radarr/Lidarr and media players can access those files.

[MichaIng]

Is there any great negative impact when enabling support for public trackers by default? I mean when adding torrents, one chooses the tracker anyway, right? I just fear that when we disable this by default we could get user reports/requests about failing connections when they use public trackers.

[MichaIng]

PR opened: #3426

[bbsixzz]

I'm currently off-grid but will leave a detailed response to your comments tomorrow.

[bbsixzz]

network.max_open_files.set we use 50 for this currently. I think 600 is way to much for a SBC? Probably we should use max_downloads.global + max_upload.global here?

I've had no issues on my two RPi4's 4GB (yet) as I'm only seeding ~250 on each machine, but I think max_downloads.global + max_upload.global makes a lot of sense.

pieces.hash.on_completion.set = no

I would strongly recommend to keep this disabled, it uses a lot of CPU and is so rarely needed that it's better left for manual right click and "Force Recheck" in ruTorrent. If many large torrents are downloading at the same time and this gets triggered the machine will become terribly slow and make other processes suffer for what is really a very unnecessary precaution.

What do you think about keeping the ration settings in place or not?

I think seeding should be left indefinitely until torrent is removed, this is best practice in the torrent community. It is considered cheap to do anything else, if the user is really concerned about some torrent running away they can lower the upload speed or manually stop the respective torrent.

protocol.encryption.set

This is really not protecting anyone, you can read more about it but to be frank it's nice as it it. You don't want to enforce encryption because it will limit your download speed because lack of peers. The way it's set up now is that it will try and get encrypted peers, then fill the rest as needed.

Lastly, you can enable public trackers because most (decent) private trackers have a private flag for their torrents so they're not affected by the client settings concerning public trackers and DHT/PEX etc.

I've read through everything you've written so these are my only comments.

I'm eagerly awaiting #3285 now :]

[MichaIng]

@bbsixzz

I would strongly recommend to keep this disabled, it uses a lot of CPU and is so rarely needed that it's better left for manual right click and "Force Recheck" in ruTorrent. If many large torrents are downloading at the same time and this gets triggered the machine will become terribly slow and make other processes suffer for what is really a very unnecessary precaution.

Okay, then we'll keep this disabled by default 👍.

I think seeding should be left indefinitely until torrent is removed, this is best practice in the torrent community. It is considered cheap to do anything else, if the user is really concerned about some torrent running away they can lower the upload speed or manually stop the respective torrent.

Okay, same as above 👍.

This is really not protecting anyone

I am just thinking that not all users use a VPN (although they should), and without VPN encryption is required to not send plain text data, isn't it?

Lastly, you can enable public trackers because most (decent) private trackers have a private flag for their torrents so they're not affected by the client settings concerning public trackers and DHT/PEX etc.

Finally it is about which trackers/torrents the user adds, right? So enabling public trackers indeed makes it just easier for users to add those, if they want it, I agree. I just don't know if those flags produce some overhead (additionally enabled features) within rTorrent? However should be marginal anyway.

[bbsixzz]

As I understand it torrent encryption does not protect your privacy, it only enables torrents to function on networks that block torrent traffic. The wording gives a false impression of security and anonymity; bottom line is if you don't have VPN you can easily be identified. Please correct me if I'm wrong.

You should enable public trackers, I don't think there's any issue with having that as default.

[MichaIng]

@bbsixzz
Any suggestion which DHT port we should use by default, if we enable it? No idea if/which port it uses when dht.port.set is no set.
EDIT: Okay 6881 is the default anyway, so we can leave this commented: https://manpages.debian.org/buster/rtorrent/rtorrent.1.en.html#DHT-RELATED_SETTINGS

I added the changes according to your suggestions: 6ee8ac1
Thanks for those 👍.

[vgarant]

@bbsixzz

system.daemon.set = true
Run as a daemon, no need to change unless for testing purposes.

rtorrent version 0.9.7+ has a built-in daemon mode but the rtorrent version included in DietPi v6.29.2 is v0.9.6.
I can't start rtorrent with this setting enabled. Can this version run in daemon mode?

[MichaIng]

@vgarant
It depends on the Debian distro. On Stretch v0.9.6 is installed and daemon mode is not enabled (since not available as you noted), but rtorrent started through a GNU screen session instead. Since Debian Buster, v0.9.7 is installed and the daemon mode is hence enabled there to allow skipping the GNU screen overhead.

[bbsixzz]

A follow up on network.port_open.set = no which can cause some private trackers to report Tracker: [Failure reason "Invalid key: port"]. This is behind a VPN.

I received the following answer from the staff:

That error happens when the URL received by the tracker announce from the torrent client is either missing the value for the mandatory port parameter, or the value is invalid. Unfortunately, there is nothing we can do to help here, it looks like something is screwing with your announce traffic.

You could try to see if your client has debug logs available and inspect those.

I have since switched to network.port_open.set = yes and everything works fine.

Not sure if you want to change the default, but I'm dropping this here for posterity.

[MichaIng]

At least we should document it: #548

It guess it is always required (respectively not set to no) when using a tracker which enforces seeding and hence requires to know at which port peers can connect? So this goes along with opening/forwarding that port or port range etc?

So probably we should extend the docs by a general seeding setup section, where we explain this + the port range options, that this is where external peers connect through and hence needs to be open/forwarded. I'm still not yet sure how exactly seeding works when no port is open, but it works, though takes often long before peers do connect, obviously by connections initiated by the seeding server. So while this may be preferred for privacy and security reasons, if either the tracker enforces it, or a higher seeding rate is required, the above setting needs to be set + port range defined and opened/forwarded accordingly.

[bbsixzz]

I'm using some of the largest private trackers that run fine with the port not open but just because they seem to tolerate it doesn't mean it's best practice, I can't explain it either but at least we know what works and what doesn't.

I don't have network.port_range.set in my config because ruTorrent defaults to 6881-6999.

I have network.port_random.set = no just because it's one less thing to worry about.

[MichaIng]

I think all other BitTorrent software listen by default on that port range, or at least one fixed port. I'm not 100% sure about the pros/cons of randomisation, but we could also set a single port only, so as well a single port needs to be opened in the router only.

I just checked:

• Transmission: G_CONFIG_INJECT '"port-forwarding-enabled"' ' "port-forwarding-enabled": true,' /etc/transmission-daemon/settings.json '^\{$'
• qBittorrent: Connection\PortRangeMin=6881
• Deluge: https://github.com/MichaIng/DietPi/blob/dev/.conf/dps_45/deluge.conf#L51-L58

Different everywhere, so it makes sense to align it, and my suggestion is:

• Enable listening on open/forwarded ports by default.
• Use the single port 6881 only by default. Not sure what the benefit of multiple ports is, but that way its easier to setup in router and monitor traffic, setup firewalls etc.
• No randomisation, which makes no sense in case of a single port anyway.
• Disable UPnP by default, which is currently mostly enabled, but IMO it is usually a security issues, as not authenticated.
• DHT enabled on same port 6881.

But what still must be made clear is that when using a VPN, incoming connections won't work anyway, as most VPN providers do not support port forwarding. So in that case it makes sense to disable port listening (to avoid confusion and overhead) and naturally some private trackers cannot be used.