transmission 2.92: doesnt support Humble Bundle torrents

[sandrotosi]

Hello,
this has been reported in Debian as http://bugs.debian.org/862063:

While attempting to download a purchase from Humble Bundle, which provides .torrent download options, I stumbled across an issue triggered by the torrent file for MiniMetro. transmission-daemon's directory watch feature failed to load the file with a metadata error. Attempting to use transmission-show resulted in the same error:

[2017-05-07 19:40:24.417] dinosaurpoloclub/MiniMetro-gamma19a-linux.tar.gz: Invalid metadata entry "path"
Error parsing .torrent file "MiniMetro-gamma19a-linux.tar.gz.torrent"

I was able to open the file using the Python library "bencode", and saw that there was no "path" metadata entry, and that the torrent was for a single file with the info/name entry "dinosaurpoloclub/MiniMetro-gamma19a-linux.tar.gz".

Removing "dinosaurpoloclub/" from that entry was enough to allow the torrent to work. I did a bit of searching, and it looks like having a full path in the info/name entry is valid (although I've never seen it before, bep 3 doesn't seem to prohibit it)

It may also be worth noting that the "dinosaurpoloclub" folder did not exist on my computer

+

As a clarification - editing the file allowed it to be loaded into transmission successfully. The download could not continue and logged the bug below. If I'm reading the standard correctly, the no-longer-matching hash of the info dictionary caused this.

MiniMetro-gamma19a-linux.tar.gz Tracker error: "unregistered torrent" (torrent.c:581)

+

I had also submitted a ticket to Humble Bundle about the torrent file. They tested it with uTorrent and it sounds like it's working there:

Hi there!

I tested the torrent and was able to download it — I don't have a Linux machine available to test at the time, but what happens when you try to install it?

I successfully torrented the file using uTorrent. Maybe you could give that client a try? You can also download the files directly via the direct downloads button — this game isn't too big, so that might be a good option.

I'm happy to look into the torrent file further, but it might take a while to get a solution for you.
Hopefully one of my proposed solutions is sufficient for you, but if you need me to look into this further, let me know and I'm happy to do so!

-Dan
Humble Bundle

[sandrotosi]

@hauntingEcho

[cfpp2p]

https://trac.transmissionbt.com/ticket/5517#comment:34

Could you please point me to at least one (or better more than one) real torrent with such structure, and/or at least one real torrent-making program which creates torrents with such structure? I'll be glad to reconsider.
https://trac.transmissionbt.com/ticket/5517#comment:35

cfpp2p/transmission@10fa7bd

[sandrotosi]

i'm afraid the only way to get such a torrent is by buying a humble bundle - @hauntingEcho as the original submitted to the debian bug, can you provide some help here?

[cfpp2p]

294-sample-torrent

examples the problem.

[hauntingEcho]

Due to copyright issues, I don't think I can forward along the torrent itself, but cfpp2p's sample torrent does demonstrate the same issue. I do want to clarify something with the title though - the problem doesn't seem to be present in any of Humble's other torrents that I've tried, just MiniMetro. I'll contact Humble again & ask if they can provide additional details on their torrent-creation process.

[hauntingEcho]

sorry for my slow response time, this is the response I got from Humble support:

Hey again,

I apologize. We're not able to comment on the methods for compiling the torrents. I know that makes reproducing harder, and I apologize.

My best recommendation in the short-term is to use a different client (like utorrent) sorry again for the hassle!

Take care, and happy gaming,

-Dan
Humble Bundle

[Astara]

Supporting an absolute path in a torrent could easily be seen as a security risk. Utilities for file transfer (like 'tar', for example), deliberately strip off absolute paths due to the security problems.

It would be all to easy to have a torrent overwrite all the files in your home directory or any directory on the system where you have write access. It makes sense to limit a file-transfer util (like transmission) to saving files under "some" subdirectory. To give torrents the ability to go out of its configured subdirectory would be a disaster waiting to happen.

[hauntingEcho]

@Astara - while going up the folder tree should definitely be prevented, the torrent still should do something other than fail entirely.

One possibility would be to just strip the path entirely as you mention. Another option would be the way zip works:

[cfpp2p]

@Astara @hauntingEcho

The original poster states:

and that the torrent was for a single file with the info/name entry "dinosaurpoloclub/MiniMetro-gamma19a-linux.tar.gz"

so this particular problem is not about supporting absolute paths, or relative paths and directory traversal security, but about the "/" path element contained in the info/name entry dinosaurpoloclub/MiniMetro-gamma19a-linux.tar.gz

This can be dealt with without security issues and without stripping the path entirely by properly recognizing valid path elements. This has been already discussed quite thoroughly in TRAC - Don't create or add torrents with "../" at the beginning of the path or "/../" anywhere in the path

[Astara]

If you strip the path, you would have people complaining because of filename collisions.

I don't see the harm in adding (on linux, anyway), "." at the front of the path, so /etc/passwd, would become ./etc/passwd -- a subdir to be created under the curdir.

That said, I'm guessing it's a matter of resources and how often this bug comes up "in general", while every bug that one reports is impacting the reporter (or they wouldn't have reported it), how many people it impacts is another matter. Most torrent creation software, I'm guessing strips out absolute path components when the torrent is created, but there are always some creation clients that won't test inputs and allow garbage-in, creating garbage-out. ;^/

Am wondering in your case, are the torrents created by humble utilities, or do they just host torrents created by any third party util. If it was the former, you could lobby them to patch up absolute paths to relative, but it sounds like its the latter, which makes it harder to get them to change things -- unless you can convince them of how it is a security risk. To do that, you might need to create a torrent that would overwrite some homedir or profile dir file as a proof of concept to convince them, though they still might abdicate responsibility saying it would be some torrent client (µtorrent?) that allows the overwrite that is at fault (i.e. engage in finger pointing). Almost every security problem is a
convergence of multiple effects from different sources that hit some 'edge case'....so finger pointing is not, unfortunately, rare.

[Astara]

@cfpp2p -- looking at the trac issue you mention shows that the path "dinosaurpoloclub/MiniMetro-gamma19a-linux.tar.gz" did work, but it looks like it was disabled due to problems on the mac(?). I've seen more than one change go into xmission to dumb it down for Mac clients, like extra sleep cycles added during torrent verification so as to not overwhelm the delicate i/o subsystem on a mac. They were reduced from what they were, but 2.92 still has some of them (verify.c#39, #139).

What I don't understand is how torrents with multiple levels of subdirs work if paths w/subdirs doesn't work. It's not uncommon to have a top-folder with source material and a subdir with extra docs or images. So what is different about the torrent you mention? Maybe the subdir has to be "pre-mentioned" as a dir in the torrent file? Yeah, sounds like the torrent file is ill-formed. I bet you could recreate the torrent file, adding the metadata element, and it would work correctly.

From the error message you got, it complained that "MiniMetro-gamma19a-linux.tar.gz" didn't match any registered torrent -- and it doesn't, as the file was under "dinosaurpoloclub/".

BTW, it might help if you 'xz' (compressed) posted the torrent file as a sample (the torrent files shouldn't have any personally identifying information, since they get distributed to anyone who wants to download the torrent).

NOTE: please only post a torrent file for free material (i.e. not for a copyrighted work). Either that or edit the filenames/dirnames to not be recognizable.

[avh4]

I just tested with all games in my Humble Bundle account with Transmission 2.92 (14714), Mac OS.

7 of the 23 games in my account failed to open in Transmission with Invalid metatdata entry "path".

[cfpp2p]

@Astara

what is different about the torrent you mention?

The difference is that transmission currently does not allow the "/" character to be contained in any torrent metadata path component, that's what triggers the torrent as invalid.

If the info dictionary has any changes made to it the hash changes so that's why you just can't change the name or such, ... didn't match any registered torrent ...

[lhriley]

Just found that this happens with the Cuphead OST (MP3 and FLAC) sold via Humble Bundle.

[2017-10-20 17:25:32.510] studiomdhr_H8o9C/Cuphead_OST_Mstrd_Aug_31_MP3.zip: Invalid metadata entry "path"
[2017-10-20 17:26:14.292] studiomdhr_H8o9C/Cuphead_OST_Mstrd_Aug_31_Flac.zip: Invalid metadata entry "path"

[cweiske]

I have the same problem with the Humble Bundle "Overgrowth" torrent.

Filename in there also has a slash: wolfiregames/overgrowth-1.0.2_build-4665-linux64.zip

[Humanimaliberation]

Same problem here.
Transmission works with a torrent file for Xubuntu 17.10 64bit
but it doesn't work with a torrent file for TwoWorlds Epic Edition

[Enverex]

As this is still an ongoing issue, I thought I'd add...

That said, I'm guessing it's a matter of resources and how often this bug comes up "in general", while every bug that one reports is impacting the reporter (or they wouldn't have reported it), how many people it impacts is another matter.

This affects almost every Humble Bundle game torrent (as in the ones that Humble distribute themselves). Only very rarely have I had one of their torrents actually work in Transmission due to this issue.

[cfpp2p]

Only very rarely have I had one of their torrents actually work in Transmission due to this issue.

ahh, but with cfpp2p/transmission@10fa7bd (plus a few other miniscule tweaks committed later for other TRAC 5517 related problems) fork of transmission with Humble Bundle and other similar, actually not a problem in transmission.

[Oblomov]

It would seem that at least the version in Debian unstable (2.94 d8e60ee) is still affected by this issue. I have over 150 Humble Bundle torrents I cannot download because of the path component in the file name —and it's not clear why the torrent specifying a subdir where to place files should be invalid, provided it's not an absolute path.

[virtual812]

Attempting to download "Outcast 2" from Humble bundle recently i ran into the issue.
My situation is a tiny bit different, i'm running a Synology NAS with the 'Download Station' app for all forms of downloading.
This download station is AFAIK just a UI for Transmission.
The response from Humble Bundle above was disappointing considering this is likely to affect many users, only a few of whom will follow it up as an issue.
For the torrent in my case it was ONLY offered by torrent and no HTTP option was available.
Happy to provide any log data, i do have shell access to the NAS, but not sure exactly what i might be looking for.

[lhriley]

This isn't a solution for Transmission, but as a workaround I am now running Deluge in a FreeNAS jail for Humble Bundle torrents with this issue. Deluge has no problem parsing these torrents.

[amagnolo]

I opened a ticket with Humble Bundle to inform them of the problem. They seem receptive, in fact they replied

I cannot guarantee any immediate changes, but I will certainly forward your feedback along to the rest of the team for further consideration!

I think that if they see that many of their customers encounter this problem, they will more likely solve it. So I suggest you write to Humble Bundle a polite request to make their torrents compatible with Transmission (include a link to this issue so they can quickly understand the matter).

[hauntingEcho]

@amagnolo - while that will solve one source of torrents which trigger this bug, the torrents that Humble are producing are valid torrents per the specification. The real bug here is that transmission does not function for single-file torrents specifying that their file should be in a child folder (which, though arguably dumb, is still valid per my understanding of the spec).

[grinapo]

@amagnolo Just don't get started on a nice dream. I've told HB about this possibly 2 years ago first, and they promised to do something about it. Then re-notified them a year ago, and they said they surely will do something about it. (They also noted that the torrents are prepared and served by an external contractor who is not cooperating, or in other words sticking to that it's all good and shiny as it is.)
I also would like to second @hauntingEcho's sentiment that they are valid, and while I see the security intent it should be possible to override since it's arbitrarily violating the specs.

[amagnolo]

OK, so it should be solved by Transmission, it would be awesome; still, I think that if HB gets enough complaints, they may be more willing to not ignore the problem that their customers are facing.

[Ambrevar]

Any update on this?

[Oblomov]

@cfpp2p do you think you could create a PR for your changesets that improve the path parsing?

As an alternative, an option to disregard any path component in the torrent (just use the basename of the path) could be useful to support these torrents.

[pyroscope]

It is called info.name and not info.path for a reason – these metafiles are garbage, and GIGO applies.

The correct way to create a foo/bar.zip metafile is to create a "multi" metafile with just one entry in info.files.

[Ziktofel]

Reproduced with Tacoma (Humble Bundle, linux version) which is now free atm (Transmission 2.92)

[hauntingEcho]

@pyroscope - going by BEP 3:

info dictionary

The name key maps to a UTF-8 encoded string which is the suggested name to save the file
(or directory) as. It is purely advisory.

and further down:

For the purposes of the other keys, the multi-file case is treated as only having
a single file by concatenating the files in the order they appear in the files list.
The files list is the value files maps to, and is a list of dictionaries containing
the following keys:

length - The length of the file, in bytes.

path - A list of UTF-8 encoded strings corresponding to subdirectory names,
the last of which is the actual file name (a zero length list is an error case).

In the single file case, the name key is the name of a file, in the muliple file case,
it's the name of a directory.

while I don't agree that having a name including paths is necessarily against spec, the spec does allow for it to be munged a bit - for example, dropping everything up to the last folder-separator would be a valid thing to do if otherwise protecting from directory traversal is not worthwhile.

Further, the fact that path only appears in the multi-file torrent spec implies to me that the correct way to add a single file in a subfolder would be solely via the name key (although I agree on principal that having a single file in a subfolder doesn't make sense)

[skyegecko]

@pyroscope brings up an interesting point in the linked discussion for rtorrent:

If you want evidence beyond the spec, the original BT client has validation checks that reject / in name.

Since the spec can be interpreted either way, it's perhaps better to refer to the closest thing there is to a reference implementation.

[Oblomov]

Juding from e.g. the source code of the original BT client as found here, the info.name path check was actually commented out at some point, and in fact slashes in path names weren't allowed even in info.files by default, which partly invalidates @pyroscope's point.

That being said, even given that the first releases of the original BT client didn't support it, the fact remains that currently in the ecosystem there are clients that produce info.names with paths, and clients that accept it. Leaning on the side of Postel's law:

be conservative in what you produce, but liberal in what you accept

I think it's better for the users if transmission, while never producing such torrents, would accept it, at least given an appropriate “strip paths” option, that would eliminate all security issue and, being opt-in for the user, give them the possibility, if they so wish, to obtain the torrent data.

[Elinvention]

Is it possible to manually fix these torrents somehow, so that they are accepted by transmission?

[hauntingEcho]

as noted in the original issue report, that will break the hash of the info dictionary & cause trackers/peers to think your torrent has been tampered with

edit with copy/paste:

As a clarification - editing the file allowed it to be loaded into transmission successfully. The download could not continue and logged the bug below. If I'm reading the standard correctly, the no-longer-matching hash of the info dictionary caused this.

MiniMetro-gamma19a-linux.tar.gz Tracker error: "unregistered torrent" (torrent.c:581)

[grinapo]

Is it possible to manually fix these torrents somehow, so that they are accepted by transmission?

You either patch transmission source to disable the check or ue a different client (like btdownload which is written in python and easier to modify).

As noted already: this check could be trivially made a command line switch, it is not a technical question but a philosophical one for the developer(s). Right now they seem to believe that they wouldn't accept that some users are actually knowledgeable enough to be able to decide when to override the check. (The check could be disabled, or slashes could be replaced by underscores, or paths could be stripped altogether [possibly that'd the worst in case of multiple directories and clashing filenames].)

[mikedld]

Thanks to everyone involved. If someone is okay with trying the latest master build, please do and report back if you notice any issues with the fix.