[WIP] Add an experimental DHT/Bittorrent probe to miniooni #986
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
DHT outputExample report:Example output: |
Bittorrent outputExample report:Example output:$ ./miniooni -n -i "magnet:?xt=urn:btih:63ACA5B1F17FD71F866BB97E09BC85FE67C880F4&dn=Anarchism+and+Other+Essays+by+Emma+Goldman+PDF&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969&tr=udp%3A%2F%2Ftracker.pirateparty.gr%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.cypherpunks.ru%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.port443.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2750%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2720%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2750%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.si%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=http%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fcoppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337%2Fannounce" bittorrent [ 0.000033] Current time: 2022-11-22 22:43:56 CET [ 0.000077] miniooni home directory: $HOME/.miniooni [ 0.000210] Looking up OONI backends; please be patient... 2022/11/22 22:43:56 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details. [ 0.078944] sessionresolver: http3://dns.google/dns-query... ok [ 0.170024] session: using probe services: {Address:https://api.ooni.io Type:https Front:} [ 0.170039] Looking up your location; please be patient... [ 0.170063] iplookup: using stun_google [ 0.222036] sessionresolver: http3://dns.google/dns-query... ok [ 0.307743] sessionresolver: http3://dns.google/dns-query... ok [ 0.307935] - country: FR [ 0.307944] - network: Association "Gitoyen" (AS20766) [ 0.307950] - resolver's IP: 172.217.33.132 [ 0.307955] - resolver's network: Google LLC (AS15169) [ 0.307989] [1/1] running with input: magnet:?xt=urn:btih:63ACA5B1F17FD71F866BB97E09BC85FE67C880F4&dn=Anarchism+and+Other+Essays+by+Emma+Goldman+PDF&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.torrent.eu.org%3A451%2Fannounce&tr=udp%3A%2F%2Fpublic.popcorn-tracker.org%3A6969&tr=udp%3A%2F%2Ftracker.pirateparty.gr%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.cypherpunks.ru%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.vanitycore.co%3A6969%2Fannounce&tr=udp%3A%2F%2Fexplodie.org%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.port443.xyz%3A6969%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2750%2Fannounce&tr=udp%3A%2F%2F9.rarbg.to%3A2720%2Fannounce&tr=udp%3A%2F%2F9.rarbg.me%3A2750%2Fannounce&tr=udp%3A%2F%2Fopen.demonii.si%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=http%3A%2F%2Ftracker.openbittorrent.com%3A80%2Fannounce&tr=udp%3A%2F%2Fopentracker.i2p.rocks%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.internetwarriors.net%3A1337%2Fannounce&tr=udp%3A%2F%2Ftracker.leechers-paradise.org%3A6969%2Fannounce&tr=udp%3A%2F%2Fcoppersurfer.tk%3A6969%2Fannounce&tr=udp%3A%2F%2Ftracker.zer0day.to%3A1337%2Fannounce [ 0.308078] Using temporary directory /tmp/ooni1447389566 [ 0.318727] Resolving DNS for tracker.coppersurfer.tk [ 0.318903] Finished DNS for tracker.coppersurfer.tk: [31.14.40.30] [ 0.319008] Resolving DNS for tracker.coppersurfer.tk [ 0.319140] Finished DNS for tracker.coppersurfer.tk: [31.14.40.30] [ 0.319173] Resolving DNS for tracker.opentrackr.org [ 0.319272] Finished DNS for tracker.opentrackr.org: [93.158.213.92] [ 0.319418] Resolving DNS for tracker.opentrackr.org [ 0.319501] Finished DNS for tracker.opentrackr.org: [93.158.213.92] [ 0.319537] Resolving DNS for tracker.torrent.eu.org [ 0.319644] Resolving DNS for tracker.coppersurfer.tk [ 0.319678] Resolving DNS for tracker.openbittorrent.com [ 0.319705] Finished DNS for tracker.torrent.eu.org: [91.216.110.52 2a03:7220:8083:cd00::1] [ 0.319758] Finished DNS for tracker.coppersurfer.tk: [31.14.40.30] [ 0.319849] Resolving DNS for opentracker.i2p.rocks [ 0.319881] Resolving DNS for tracker.coppersurfer.tk [ 0.319893] Finished DNS for tracker.openbittorrent.com: [45.154.253.8 45.154.253.5 45.154.253.7 45.154.253.9 45.154.253.4 45.154.253.6 45.154.253.10] [ 0.319941] Finished DNS for opentracker.i2p.rocks: [23.137.251.45] [ 0.319966] Finished DNS for tracker.coppersurfer.tk: [31.14.40.30] [ 0.320046] Resolving DNS for opentracker.i2p.rocks [ 0.320059] Resolving DNS for tracker.torrent.eu.org [ 0.320117] Finished DNS for opentracker.i2p.rocks: [23.137.251.45] [ 0.320171] Finished DNS for tracker.torrent.eu.org: [91.216.110.52 2a03:7220:8083:cd00::1] [ 0.320217] Resolving DNS for tracker.internetwarriors.net [ 0.320315] Resolving DNS for public.popcorn-tracker.org [ 0.320354] Finished DNS for tracker.internetwarriors.net: [186.67.244.96] [ 0.320439] Finished DNS for public.popcorn-tracker.org: [::1 127.0.0.1] [ 0.320461] Resolving DNS for tracker.internetwarriors.net [ 0.320482] Resolving DNS for tracker.zer0day.to [ 0.320534] Finished DNS for tracker.internetwarriors.net: [186.67.244.96] [ 0.320555] Resolving DNS for public.popcorn-tracker.org [ 0.320584] Finished DNS for tracker.zer0day.to: [185.53.177.53] [ 0.320614] Resolving DNS for tracker.leechers-paradise.org [ 0.320664] Finished DNS for public.popcorn-tracker.org: [::1 127.0.0.1] [ 0.320698] Finished DNS for tracker.leechers-paradise.org: [199.59.243.222] [ 0.320805] Resolving DNS for tracker.leechers-paradise.org [ 0.320882] Finished DNS for tracker.leechers-paradise.org: [199.59.243.222] [ 0.320892] Resolving DNS for tracker.pirateparty.gr [ 0.320919] Resolving DNS for tracker.cypherpunks.ru [ 0.321900] Resolving DNS for tracker.cypherpunks.ru [ 0.322060] Resolving DNS for coppersurfer.tk [ 0.322190] Finished DNS for coppersurfer.tk: [31.14.40.31] [ 0.322416] Resolving DNS for coppersurfer.tk [ 0.320905] Resolving DNS for tracker.pirateparty.gr [ 0.324357] Finished DNS for coppersurfer.tk: [31.14.40.31] [ 0.324573] Resolving DNS for 9.rarbg.me [ 0.324735] Resolving DNS for tracker.vanitycore.co [ 0.324830] Resolving DNS for tracker.zer0day.to [ 0.325044] Finished DNS for tracker.zer0day.to: [185.53.177.53] [ 0.325224] Resolving DNS for open.demonii.si [ 0.325473] Resolving DNS for 9.rarbg.to [ 0.325755] Finished DNS for open.demonii.si: [46.8.8.100] [ 0.325799] Resolving DNS for 9.rarbg.to [ 0.325842] Finished DNS for 9.rarbg.to: [151.80.120.112 151.80.120.115 151.80.120.113 151.80.120.114] [ 0.325950] Resolving DNS for tracker.vanitycore.co [ 0.325979] Finished DNS for 9.rarbg.to: [151.80.120.112 151.80.120.115 151.80.120.113 151.80.120.114] [ 0.326016] Resolving DNS for 9.rarbg.to [ 0.326538] Finished DNS for 9.rarbg.to: [151.80.120.112 151.80.120.115 151.80.120.113 151.80.120.114] [ 0.326598] Resolving DNS for 9.rarbg.to [ 0.326786] Finished DNS for 9.rarbg.to: [151.80.120.112 151.80.120.115 151.80.120.113 151.80.120.114] [ 0.327707] Resolving DNS for tracker.port443.xyz [ 0.328229] Resolving DNS for explodie.org [ 0.328326] Resolving DNS for tracker.port443.xyz [ 0.328402] Finished DNS for explodie.org: [184.105.151.166 2001:470:1:189:0:1:2:3] [ 0.328492] Finished DNS for 9.rarbg.me: [151.80.120.114 151.80.120.113 151.80.120.112 151.80.120.115] [ 0.328523] Resolving DNS for 9.rarbg.me [ 0.328531] Resolving DNS for open.demonii.si [ 0.328551] Resolving DNS for explodie.org [ 0.328605] Finished DNS for open.demonii.si: [46.8.8.100] [ 0.328625] Finished DNS for explodie.org: [184.105.151.166 2001:470:1:189:0:1:2:3] [ 0.328635] Finished DNS for 9.rarbg.me: [151.80.120.114 151.80.120.113 151.80.120.112 151.80.120.115] [ 0.780205] Resolving DNS for open.demonii.si [ 0.780321] Resolving DNS for tracker.port443.xyz [ 0.780357] Resolving DNS for tracker.torrent.eu.org [ 0.780379] Resolving DNS for tracker.leechers-paradise.org [ 0.780407] Resolving DNS for 9.rarbg.to [ 0.780506] Finished DNS for tracker.leechers-paradise.org: [199.59.243.222] [ 0.780515] Finished DNS for tracker.torrent.eu.org: [91.216.110.52 2a03:7220:8083:cd00::1] [ 0.780527] Resolving DNS for tracker.zer0day.to [ 0.780537] Resolving DNS for opentracker.i2p.rocks [ 0.780552] Resolving DNS for 9.rarbg.to [ 0.780590] Finished DNS for tracker.zer0day.to: [185.53.177.53] [ 0.780596] Finished DNS for opentracker.i2p.rocks: [23.137.251.45] [ 0.780286] Resolving DNS for coppersurfer.tk [ 0.780657] Finished DNS for coppersurfer.tk: [31.14.40.31] [ 0.780303] Resolving DNS for tracker.vanitycore.co [ 0.780252] Resolving DNS for tracker.opentrackr.org [ 0.780696] Resolving DNS for tracker.pirateparty.gr [ 0.780713] Resolving DNS for tracker.coppersurfer.tk [ 0.780731] Finished DNS for tracker.opentrackr.org: [93.158.213.92] [ 0.780262] Resolving DNS for tracker.cypherpunks.ru [ 0.780241] Resolving DNS for explodie.org [ 0.780764] Finished DNS for tracker.coppersurfer.tk: [31.14.40.30] [ 0.780606] Resolving DNS for tracker.internetwarriors.net [ 0.780264] Resolving DNS for 9.rarbg.me [ 0.780817] Finished DNS for tracker.internetwarriors.net: [186.67.244.96] [ 0.780832] Resolving DNS for tracker.opentrackr.org [ 0.780200] Resolving DNS for explodie.org [ 0.780863] Finished DNS for 9.rarbg.me: [151.80.120.114 151.80.120.113 151.80.120.112 151.80.120.115] [ 0.780398] Resolving DNS for tracker.pirateparty.gr [ 0.780884] Finished DNS for tracker.opentrackr.org: [93.158.213.92] [ 0.780283] Resolving DNS for 9.rarbg.to [ 0.780389] Finished DNS for open.demonii.si: [46.8.8.100] [ 0.780894] Resolving DNS for tracker.coppersurfer.tk [ 0.780247] Resolving DNS for 9.rarbg.me [ 0.780959] Finished DNS for explodie.org: [184.105.151.166 2001:470:1:189:0:1:2:3] [ 0.780975] Finished DNS for tracker.coppersurfer.tk: [31.14.40.30] [ 0.780537] Finished DNS for 9.rarbg.to: [151.80.120.112 151.80.120.115 151.80.120.113 151.80.120.114] [ 0.780277] Resolving DNS for 9.rarbg.to [ 0.781046] Finished DNS for 9.rarbg.me: [151.80.120.114 151.80.120.113 151.80.120.112 151.80.120.115] [ 0.780989] Finished DNS for 9.rarbg.to: [151.80.120.112 151.80.120.115 151.80.120.113 151.80.120.114] [ 0.780296] Resolving DNS for tracker.port443.xyz [ 0.780310] Resolving DNS for tracker.vanitycore.co [ 0.780258] Resolving DNS for opentracker.i2p.rocks [ 0.781155] Finished DNS for opentracker.i2p.rocks: [23.137.251.45] [ 0.780271] Resolving DNS for tracker.torrent.eu.org [ 0.780692] Finished DNS for 9.rarbg.to: [151.80.120.112 151.80.120.115 151.80.120.113 151.80.120.114] [ 0.780844] Finished DNS for explodie.org: [184.105.151.166 2001:470:1:189:0:1:2:3] [ 0.780255] Resolving DNS for tracker.cypherpunks.ru [ 0.781239] Finished DNS for tracker.torrent.eu.org: [91.216.110.52 2a03:7220:8083:cd00::1] [ 0.781278] Finished DNS for 9.rarbg.to: [151.80.120.112 151.80.120.115 151.80.120.113 151.80.120.114] [ 0.785650] Resolving DNS for tracker.internetwarriors.net [ 0.785672] Resolving DNS for public.popcorn-tracker.org [ 0.785719] Finished DNS for tracker.internetwarriors.net: [186.67.244.96] [ 0.785743] Resolving DNS for public.popcorn-tracker.org [ 0.785822] Finished DNS for public.popcorn-tracker.org: [::1 127.0.0.1] [ 0.785849] Resolving DNS for tracker.leechers-paradise.org [ 0.785865] Resolving DNS for coppersurfer.tk [ 0.785900] Finished DNS for tracker.leechers-paradise.org: [199.59.243.222] [ 0.785851] Finished DNS for public.popcorn-tracker.org: [::1 127.0.0.1] [ 0.785982] Resolving DNS for tracker.zer0day.to [ 0.786014] Resolving DNS for open.demonii.si [ 0.786003] Resolving DNS for tracker.openbittorrent.com [ 0.786071] Resolving DNS for tracker.coppersurfer.tk [ 0.786107] Resolving DNS for tracker.coppersurfer.tk [ 0.786974] saving measurement to disk [ 0.787212] experiment: recv 0.00 byte, sent 0.00 byte [ 0.787371] sessionresolver: [{"URL":"http3://dns.google/dns-query","Score":0.999999999999999},{"URL":"https://dns.google/dns-query","Score":0.999999999990001},{"URL":"https://cloudflare-dns.com/dns-query","Score":0.1},{"URL":"http3://cloudflare-dns.com/dns-query","Score":1.3065307323292716e-8},{"URL":"https://mozilla.cloudflare-dns.com/dns-query","Score":1.0000000000000005e-8},{"URL":"http3://mozilla.cloudflare-dns.com/dns-query","Score":9.999999999999979e-9},{"URL":"https://dns.quad9.net/dns-query","Score":9.996899376120266e-9},{"URL":"system:///","Score":0}] [ 0.787418] whole session: recv 7.23 kbyte, sent 2.06 kbyte |
Test resultsThe following test results are accompanied by tons of build fails of everything that depends on sqlite. See the TODO above for context. |
|
Continuing investigation for the SQLite issue. I can reproduce on a different system (archlinux) it in the following steps:
I'm afraid i'm going to need someone who is much more experienced with the golang module system than i am to fix this one. |
|
Continuing investigation for the SQLite issue... It seems i can get around the golang panic by:
Now i run into linking issues due to duplicate definitions: This can be worked around by adding However, it's not good for the project to require manual flags handling just for a single probe. But it seems from reading around various issues regarding feature tags and build flags that go devs are very hostile to declaring default compilation flags in go.mod, instead recommending to use an external build system. But i've been thinking we don't need the sqlite features of torrent/dht in this probe. SQLite features are not even in the code paths used, so it's not clear to me why go would try to compile this second copy of SQLite in the first place. I've been trying to fork anacrolix modules to remove the hard-coded SQLite dependency by "replace"ing modules in the go.mod declaration. However, i have so far failed to get this working because:
What i have not tried so far is to fork the repo on github and use imports like Does an experienced gopher have a clue about how the module system works and what would be a simple and recommended way to go? I have read tons of documentation and tutorial about go modules but could not find what i was looking for. What i'm trying to achieve would be extra simple in C or Rust so i'm either missing the obvious (very likely), or the golang module system is entirely useless. I hope i don't sound too angry here, but i've just lost hours and hours trying to work around this problem so i'm a little sour. PS: I should mention that anacrolix/torrent also has a "nosqlite" build tag, making it possible to |
|
I have some questions about the DHT data model and what to measure precisely. For now in the PR i have two modes of operation:
I'm currently refactoring the code to make it more readable, and i'm slowly starting to think that this is not consistent. But i don't know precisely what would be interesting to measure: do we care that separate bootstrap nodes are reachable from the probe? If so, do we care what addresses were reachable (like IPv6 works but not IPv4)? Or do we just care that the DHT is globally reachable from a given list of bootstrap nodes? If we want per-IP bootstrapping feedback, should i store the DNS results globally as is currently the case? Or remove the global DNS results test key, place it in individual test keys, then have redundant DNS information for every IP matching a given hostname? Or create a new layer so that a run consists of a list of DNS queries and a list of individual DHT bootstrapping measurements per IP? One last question: when resolving the hostnames for the bootstrap nodes, i assume failing all DNS resolution should result in a global failure. However, should individual hostname resolution failure add to the failure key as well? In most other tests (eg. SMTP/IMAP) failing to resolve logically is a global failure, but in the case of DHT probing where we try a bunch of different nodes, it's not clear to me that failing to resolve a few nodes is actually a failure. It stills seems important to expose that information, so even though it's already in the DNS results key, should i add a FailedHostname key for easier parsing? |
|
I did some light testing of the bittorrent test. It compiles, it runs and it produces a JSONL file with some information inside of it. I think from the perspective of what can be expected to have built in a hackathon this fits the bill. Good job! 🥳 That said I will need some more time to provide an in depth review of how the test works, the data format it produces, how we would go about analyzing the data and what we care to measure. It's a great MVP and I think you already highlighted some of the key areas that need further work before it's ready for prime-time. |
|
@bcmills, is the above useful regarding dependency resolution failures? |
|
@anacrolix Since you're around here, how realistic would it be to remove hard-coded sqlite dependency in your packages? It's currently imported by missinggo which makes it a dependency in almost all your packages. From a quick github search it seems those helpers are not currently used in your repos so maybe they could just be removed? As for the torrent package, could sqlite (and many other storages) be moved into a separate repository/package? or does go require that you implement the storage interfaces from the same package? If that sounds possible and fine to you i'd be happy to make a PR, although if you'd like to take care of this yourself i'll leave that up to you (since it anyway requires bumping missinggo/torrent dep and go mod tidy in several repos to remove the indirect dependency completely). |
|
@anacrolix, thanks for flagging this — looks like more solid evidence for golang/go#55955 being deeply related to golang/go#56494 (and I'm actively working on a fix, but it's complicated). I don't have a good workaround at the moment except for using |
|
@ooninoob I'm happy to remove it where possible. Note that the only issue with sqlite should be the linking at the end, or any CGO compilation on platforms that don't support it. I think there are several sqlite packages floating around that link sqlite in varying manners so they should be made to defer to the final applications linking preferences for those. Please link me to any specific examples (maybe as issues in those repos so as not to clutter this issue), or submit PRs where you deem appropriate. |
|
Also sorry for the pain in using anacrolix/torrent and some of the other modules at the moment, they're very old as Go is concerned, and have very long entangled histories, and adopted Go modules very early, so they're in the sweet spot as far as triggering package dependency management bugs. |
I've taken a look. I think the crawshaw.io/sqlite dependency in anacrolix/torrent (via storage) is conflicting with your use of mattn/sqlite? I was able to get this PR to build with anacrolix/torrent#421 (comment). Specifically I ran
Could you link to those ones? I still can't find them.
I'll take a look at that one now.
I think I already covered that in #986 (comment). |
|
Looks like you can also build with |
Don't worry, it's already great that you've made such work so far, and we can work out the quirks as we go :)
Like i said, it's possible by passing CGO flags or by passing a nosqlite tag to bypass the issue. However it's not good developer UX to require passing flags manually just to get something to build... So if we can find another solution i'm interested.
I don't know much about go modules. Would it be possible to move most of the torrent stuff inside a torrent-core package, and reexpose it inside torrent with default sqlite storage? That way we could use torrent-core here in this PR and people would keep using torrent with default SQLite storage without breaking API/behavior, but torrent-core would have zero sqlite dependency. |
|
I've pushed more information in the DHT test output. I've also pushed an initial Bittorrent test which fails. From a first glance it seems related to the DHT test issue. The two nodes have back and forth DHT queries/responses but nothing seems to be done with it. Output from the Bittorrent test:
The first errors complain that the first node (the bootstrap node) has no bootstrap nodes to bootstrap the DHT with, which is intentional because we're running the test fully offline on the loopback interface. Then the second client gets started successfully with the first bootstrap node... but then the announce produces nothing. The Client from the probe seems to have found a good node for each interface (IPv4/IPv6 localhost) but it doesn't find the torrent in the announce. I've attached the Wireshark capture from the test run, please rename it to I'm wondering if maybe anacrolix/dht ignores DHT bootstrap nodes in the announce responses? That theory would explain why both DHT/Bittorrent tests fail, while both probes run fine against mainline DHT where bootstrap nodes are just that (don't actually announce/seed hashes) but more nodes are available. |
Unfortunately Go doesn't provide more advanced compilation where we can detect the presence of other sqlite implementations etc. Additionally, if I make your case with 2 sqlite repos work by default, then sqlite would necessarily be disabled by default without any tags (and I would forever be telling users to add
Both direct usages are in subpackages that should only be imported when required, so quite isolated. The other reference is a generic conversion utility type, and doesn't pull in any actual sqlite packages.
Not without losing any kind of "default" behaviour. I would need to rely on examples that do stuff like |
Try my earlier recommendation about secure nodes. Set |
Sorry i missed that part. I just tried and it produces exactly the same results. However, by forcing to use IPv4, it seems the test now pass, even without setting NoSecurity: I don't have IPv6 on my home network, but i do have IPv6 on the loopback interface (who doesn't?). I'm not sure if that's a anacrolix/dht bug or something else. To quote your package's docs, i think aliens are involved here :) I'll try and see how that affects the Bittorrent test, although it seems that the torrent package runs one DHT server per address and used both IPv4 and IPv6 in my case. I'd also be curious if someone can reproduce on their own machine that the test passes with IPv4 but fails with IPv6. EDIT: NoSecurity has no effect on the Bittorrent test, and i don't see how forcing IPv4 would help because it's already enabled in one of the DHT servers in each torrent.Client. Test output:
|
I don't think that is the case, because those subpackages use missinggo's go.mod, so it seems sqlite is declared as a "global" dependency for the missinggo import. For example, anacrolix/tagflag only requires the top-level missinggo package, does not use a specific runid subpackage, yet inherits sqlite as a dependency in its go.sum.
I'm still very unfamiliar with go, but it appears to be possible to reexport stuff from a different package while overriding certain methods. In which case anacrolix/torrent could reexport anacrolix/torrentcore types while making sure SQLite is the default storage in that case, and such a transition should be 100% transparent to API consumers (because the exposed types would still be named torrent.Whatever, not torrentcore.Whatever). Am i misunderstanding this golang pattern? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Checklist
Description
As part of the InternetBorders hackathon, i am submitting a work-in-progress pull request to add DHT/Bittorrent blocking detection support in miniooni. I was mentored by @hellais during the hackathon, who introduced me to the many flavors of ooni APIs.
I am aware that the contribution guidelines favor small patches, however these two small modules introduce the same set of dependencies and are not overtly complex to review. Please let me know if i should submit two pull requests instead.
There is still some amount of work to do before merging. I may not have followed golang best practices (i'm very new), and may not have followed ooni internal API best practices. I am ready to dedicate at least at least 3 more days of my time full-time in order to get this pull request merged, given sufficient review/guidance.
Input
The dht probe expects a
udp://ip:portaddress to try and bootstrap the DHT from. If a domain is passed as input, the domain will be resolved and the resolved IPs will be checked separately (in different Runs). If "DUMMY" is passed as input, a well-known list of bootstrap nodes will be resolved, and all resolved IPs will be attempted as part of the same run (testing overall DHT connectivity).The infohash announced by the dht probe is static and is the Bittorrentv2 hybrid test infohash (the v1 infohash of it), as presented here. It is not a suspicious torrent that may be specifically censored, or may get people into legal trouble for announcing it.
The bittorrent probe expects a magnet URL as input. Please feed it a magnet linking to small content so it will succeed before timeout. The content of the magnet will be downloaded to a temporary directory, then removed.
The torrent/dht library used only supports Bittorrentv1 at the moment (not Bittorrentv2) but i don't think that is a problem because most clients only support v1 (and i suppose the censors too, or at least would block both not just v2).
Output/Report
Github says PR body is too long so i will post sample output/report and test results in comments
DHT has the following data in the report's test keys:
Where an individual test run contains:
Bittorrent has the following data in the report's test keys:
TODO
Note about "DUMMY": i actually tried to check for empty input as a simple way of achieving this, but this produces a runner error. I think i will have to use proper static input in that case. The error, for reference, is triggered whether i don't specify -i flag, or if i use -i "":
Note about SQLite: for some reason i had to manually remove a line from go.sum to import the torrent/dht library. This line was about SQLite, and go would panic (yes panic, as described here) when trying to "go get" the library.
go mod tidydid not fix it. Manually removing that line from go.sum seemed to fix it... until i ran all tests. This issue may be related to my usage of NixOS which can produce weird linker issues. I will try to reproduce on another distribution.