Difficulty of probing Tor hidden services

[ageis]

Debian stretch/9.x
all versions of blackbox_exporter

This is sort of partly a bug and partly a feature request. I wanted to monitor some Tor hidden services using Prometheus and I eventually got it to work, but it required a lot of effort. I discovered that these tools are pretty ill-suited to that use case. Unfortunately I can't tell you exactly what the magic combination was since there's so many factors involved, but I'd like to document my experience and touch upon the difficulties encountered. I'll provide configuration files at the bottom of this post.

My setup is thus: I run Tor with the DNSPort (5353), SOCKSPort (9050) and TransPort (9040) enabled. I configure it to listen on and accept requests from localhost, as well as resolve all .onion addresses to a virtual/fake IPv4 address (AutomapHostsOnResolve+VirtualAddrNetworkIPv4). For some help with caching, I run dnsmasq and put it on top of that, and I then direct all name resolution on my machine through Tor via resolv.conf and/or iptables.

We also need to have a transparent HTTP proxy in the picture, since Tor doesn't provide one. Alas, it only provides support for SOCKS, transparent TCP proxying, and DNS. SOCKS is not supported by blackbox_exporter, so I guess that would be one of the first problems I'd highlight. But anyway, to solve that I run Privoxy, which listens on port 8118 and sends HTTP requests through Tor's SOCKS port. I set proxy_url in the params of my scrape configs in prometheus.yml and include that in blackbox.yml as well, as well as setting the HTTP_PROXY environment variable.

Right off the bat, I had issues with Go's built-in DNS resolver. We need to use CGO in order for this to work! Apparently you can force CGO by setting the RES_OPTIONS environment variable, but I'm not sure on that. So I rebuilt Go from source with the netcgo build tag, plus built the blackbox_exporter binary with GODEBUG="netdns=cgo+1" GOFLAGS="-tags=netcgo" CGO_ENABLED=1.

One other thing which I tried and should mention, yet I'm not sure to what extent it helped, is modifying the ExecStart command in prometheus-blackbox-exporter's systemd service unit and prepending /usr/bin/torsocks to it. That's a shell wrapper which is supposed to make everything go through the Tor network by using the torsocks library to re-write system calls—in theory that should mean I wouldn't need the proxy or DNS setup here. I experimented with different versions, e.g. the binary 0.9.1 package in stretch-backports vs. the current master branch and got different results. Sometimes torify/torsocks did seem to help name resolution complete successfully, other times it looked like it had no effect. Another way of doing it is like this:

Environment=LD_PRELOAD=/usr/lib/x86_64-linux-gnu/torsocks/libtorsocks.so

Throughout this whole process, I witnessed a variety of errors. "resolution with preferred IP protocol failed, attempting fallback protocol"..."temporary failure in name resolution"..."context deadline exceeded"..."connection reset by peer" (from Privoxy)..."Error resolving address...no suitable address found" and so on.

To summarize, the primary issue with probing Tor hidden services is that blackbox_exporter wants to do DNS resolution of the target address before passing the request along to the HTTP proxy, instead of letting the proxy handle it. The more problematic failure that I've seen is where resolution succeeds except blackbox_exporter reports an error (EOF) in the HTTP request: "Error for HTTP request" err="Get http://[10.197.95.211]: EOF" which is weird and kind of smells like it never passed the request along to the proxy and tried to do the probing itself. The other issue is that for some reason we can't use a binary which prefers Go's built-in resolver and need to use CGO in order to access Tor hidden services.

I doubt so many users want this, but I would suggest a new config option to help make all of this easier. Such an option would essentially disable the successful DNS resolution requirement and still pass the request along to the specified proxy regardless of any error there.

I hope this has been informative. Now, without further ado, here's my configs for anyone else who wants to experiment.

/etc/prometheus/blackbox.yml:

modules:
  http_2xx:
    prober: http
    timeout: 30s
    http:
      method: GET
      valid_status_codes: [200, 302]
      no_follow_redirects: true
      proxy_url: http://127.0.0.1:8118
      tls_config:
        insecure_skip_verify: true
      preferred_ip_protocol: "ip4"

Extra environment variables for /etc/default/prometheus-blackbox-exporter in addition to ARGS:

HTTP_PROXY="http://127.0.0.1:8118"
RES_OPTIONS="debug"
GODEBUG="netdns=cgo"
CGO_ENABLED=1

/etc/prometheus/prometheus.yml:

global:
  scrape_interval: 120s
  evaluation_interval: 30s
  scrape_timeout: 30s
  external_labels:
      monitor: 'tor'

rule_files:
  - alerts/*.alerts

scrape_configs:
  - job_name: 'tor'
    metrics_path: /probe
    params:
      module: [http_2xx]
      proxy_url: ['127.0.0.1:8118']
    static_configs:
      - targets: ['facebookcorewwwi.onion']
        labels:
          instance: 'Facebook'
      - targets: ['blockchainbdgpzk.onion']
        labels:
          instance: 'Blockchain.info'
      - targets: ['qubesos4rrrrz6n4.onion']
        labels:
          instance: 'Qubes OS'
      - targets: ['privacyintyqcroe.onion']
        labels:
          instance: 'Privacy International'
      - targets: ['3g2upl4pq6kufc4m.onion']
        labels:
          instance: 'DuckDuckGo'
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: address
      - target_label: __address__
        replacement: 127.0.0.1
# you might need to append the port :9115 right above, in my case I had a DNS name and an nginx reverse proxy listening on port 80 for prometheus-blackbox-exporter

/etc/tor/torrc:

SOCKSPort 127.0.0.1:9050 PreferSOCKSNoAuth
SOCKSPolicy accept 127.0.0.1
SOCKSPolicy reject *
TestSocks 1

Log info file /var/log/tor/info.log
Log notice file /var/log/tor/notice.log
Log debug file /var/log/tor/debug.log
Log warn file /var/log/tor/warn.log

SafeLogging 0
ControlPort 9051
CookieAuthentication 1

VirtualAddrNetworkIPv4 10.192.0.0/10
AutomapHostsOnResolve 1
AutomapHostsSuffixes .onion
TransPort 9040
TransListenAddress 127.0.0.1
DNSPort 5353
DNSListenAddress 127.0.0.1

/etc/dnsmasq.conf:

no-resolv
no-poll
port=53
server=127.0.0.1#5353
listen-address=127.0.0.1
cache-size=1024
local-ttl=3600
min-cache-ttl=3600

/etc/privoxy/config:

user-manual /usr/share/doc/privoxy/user-manual
admin-address [email protected]
confdir /etc/privoxy
logdir /var/log/privoxy
actionsfile match-all.action
actionsfile default.action
filterfile default.filter
logfile logfile
debug     13313
hostname localhost
listen-address  127.0.0.1:8118
toggle  0
enable-remote-toggle  0
enable-remote-http-toggle  0
enable-edit-actions 0
enforce-blocks 0
permit-access  localhost
buffer-limit 8192
enable-proxy-authentication-forwarding 0
forward-socks5 / 127.0.0.1:9050 .
forward-socks5t / 127.0.0.1:9050 .
forward           127.*.*.*/     .
forward           localhost/     .
forwarded-connect-retries  0
accept-intercepted-requests 0
allow-cgi-request-crunching 0
split-large-forms 0
keep-alive-timeout 5
tolerate-pipelining 1
socket-timeout 300
log-messages   1
log-max-lines 65535

[brian-brazil]

The blackbox exporter is intended to handle a small number of standard of probes. The Tor use case is quite a bit away from that on several axes, so as with anything like that I'd suggest writing a custom exporter rather than trying to overload the blackbox exporter.

[marcan]

To summarize, the primary issue with probing Tor hidden services is that blackbox_exporter wants to do DNS resolution of the target address before passing the request along to the HTTP proxy, instead of letting the proxy handle it.

Use case aside, that's a bug. I would never expect an HTTP proxy client to perform DNS resolution (of anything but the proxy's hostname itself). That's not how HTTP proxies are supposed to work.

[brian-brazil]

Our support for control of v4/v6 requires us to do the resolution ourselves. It's expected that the vast majority of users are hitting services directly, not via a proxy. That's merely something that comes in with the http library we use.

[brian-brazil]

Thinking on this a bit, our current http module isn't great for testing of HTTP proxies which is something we should really support. Similar to how the DNS module works, what would you think of a module/option where the target was the proxy to use and the actual HTTP request line coming from the config?

Have you looked at doing what you want via the TCP module?

[ageis]

@brian-brazil That sounds good to me. As long as I can pass an .onion DNS name along to the proxy without getting resolution errors from blackbox, then I can have the proxy handle SOCKS/Tor and everything else.

I've not looked at using the TCP module and not sure what that would look like... I'm seeking HTTP 200 responses, the goal is to make sure these hidden services are up. Think of an ecosystem like SecureDrop where you have 40+ of these targets. You might even want to check for a version string showing up in the page.

[darkk]

@ageis there is another way to achieve the goal of testing *.onion addresses with Prometheus.

Unfortunately, there is no clean way to resolve kavakavakavakava.onion via DNSPort from blackbox_exporter because of golang.org/issue/13705 — net.ResolveIPAddr() will ignore *.onion names.

Also, I looked at current blackbox_exporter codebase, and I consider it impractical to add Socks5 proxy support to every prober speaking over TCP. But it may also be useful for cases when the host-to-be-probbed is reachable via OpenSSH DynamicPort tunnel, not just for Onion services. Also, proxy_url is not suitable for my case as I want to check non-HTTP endpoints for liveness.

I see following extention points to solve the issue:

• adding static configuration with onion name under some other domain and do domain rewriting on the path from blackbox_exporter to DNSPort (e.g. with another DNS server)
• generate file_sd_config configuration with another program (reading onion names from some other data source) and use IPs while generating blackbox_exporter configuration
• abuse some "configurable" service-discovery API and provide a shim that converts onion names written in prometheus configuration file to VirtualAddrNetworkIPv6 addresses fetched from tor daemon

Seems, dns_sd_config fits to some extent. It queries DNS directly and neither prometheus code, nor resolver library filter DNS queries to *.onion TLD. One of missing bits is that "the DNS servers to be contacted are read from /etc/resolv.conf. But it's possible to redirect DNS queries to *.onion domains to different DNS server with BPF filter at netfilter level.

The method I describe has following flaws:

• Host header sent by blackbox_exporter contains (and leaks!) virtual IPv6 address unless overriden
• TLS certificate validation by blackbox_exporter needs server_name as target is IPv6 address
• some of well-known *.onion webservers need Host header (e.g. the onion for onion.debian.org), some don't (e.g. keybase.io), so the header has to be specified in blackbox.yml
• the specific BPF rule does not capture subdomains like www.facebookcorewwwi.onion and does not capture prop224 / Onion v3 / 56-char onion names, so those name will not be monitored and will leak to resolver
• the specific BPF rule does not work for IPv6 packets as udp is broken for IPv6, it matters if you have IPv6 nameserver in resolv.conf
• some metrics become garbage: probe_dns_lookup_time_seconds, probe_http_duration_seconds{phase="connect"}, probe_http_duration_seconds{phase="resolve"}

IMHO, pain with BPF rules should be reduced with some code implementing nameserver option for dns_sd_config, but I'm unsure what's Prometheus developers' opinion on that matter. Here is example of configuration:

prometheus.yml

---
global:
  scrape_interval:     30s # Default is every 1 minute.
  evaluation_interval: 30s # The default is every 1 minute.

scrape_configs:
  - job_name: ssh
    metrics_path: /probe
    params: {module: [ssh_banner]}
    dns_sd_configs:
      - names: [hedgeivoyioq5trz.onion] # hedgei.torproject.org
        type: AAAA
        port: 22
        refresh_interval: 60s # tor DNS TTL
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*)
        target_label: __param_target
        replacement: ${1}
      - {source_labels: [], regex: .*, target_label: __address__, replacement: 127.0.0.1:9115}
      - source_labels: [__meta_dns_name]
        regex: (.*)
        target_label: instance
        replacement: ${1}:22 # NB, port!

  - job_name: 'http'
    metrics_path: /probe
    params: {module: [http_keybase]}
    dns_sd_configs:
      - names: [fncuwbiisyh6ak3i.onion] # keybase.io
        type: AAAA
        port: 80 # port is mandatory for `dns_sd_configs`
        refresh_interval: 60s # tor DNS TTL
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*)
        target_label: __param_target
        replacement: ${1}
      - {source_labels: [], regex: .*, target_label: __address__, replacement: 127.0.0.1:9115}
      - source_labels: [__meta_dns_name]
        regex: (.*)
        target_label: instance
        replacement: ${1} # `http://` prefix and port are not needed for this example

  - job_name: 'https'
    metrics_path: /probe
    params: {module: [http_protonmail]}
    dns_sd_configs:
      - names: [protonirockerxow.onion]
        type: AAAA
        port: 443 # as it's mandatory
        refresh_interval: 60s # tor DNS TTL
    relabel_configs:
      - source_labels: [__address__]
        regex: (.*)
        target_label: __param_target
        replacement: https://${1} # NB: target name is non-trivial here
      - {source_labels: [], regex: .*, target_label: __address__, replacement: 127.0.0.1:9115}
      - source_labels: [__meta_dns_name]
        regex: (.*)
        target_label: instance
        replacement: https://${1} # without the prefix `http://` is implied
...

blackbox.yml

---
modules:
  ssh_banner:
    prober: tcp
    timeout: 5s
    tcp:
      query_response:
      # - \x0a is auto-added https://github.com/prometheus/blackbox_exporter/blob/master/tcp.go#L127
      # - blackbox_exporter waits for newline, so we can't wait for another part of handshake :-(
      - send: "SSH-2.0-blackbox_exporter prometheus-0.0\x0d"
        expect: "^SSH-2.0-"

  http_keybase:
    prober: http
    timeout: 5s
    http:
      no_follow_redirects: true
      fail_if_not_matches_regexp: ["<title>Keybase</title>"]

  http_protonmail:
    prober: http
    timeout: 5s
    http:
      no_follow_redirects: true
      fail_if_not_ssl: true # not needed, but `tls_config` is there anyway :)
      fail_if_not_matches_regexp: ["<title>Login - ProtonMail</title>"]
      headers:
        Host: protonirockerxow.onion # XXX: otherwise ProtonMail returns 400 Bad Request
      tls_config:
        server_name: protonirockerxow.onion # XXX: another duplicate of target name
...

torrc

TransPort [::1]:9099 OnionTrafficOnly NoDNSRequest
AutomapHostsOnResolve 1
AutomapHostsSuffixes . # do not use DNS at DNSPort, map everything
VirtualAddrNetworkIPv6 [fddd:4466:17aa::]/48 # Some IPv6 ULA
DNSPort 127.0.0.1:9053

iptables

iptables -t nat -I OUTPUT -p udp --dport 53 -m bpf --bytecode '23,48 0 0 0,84 0 0 240,21 0 19 64,48 0 0 9,21 0 17 17,40 0 0 6,69 15 0 8191,177 0 0 0,64 0 0 10,84 0 0 2147549183,21 0 11 1,64 0 0 14,21 0 9 0,80 0 0 20,21 0 7 16,64 0 0 36,84 0 0 16768991,21 0 4 347982,64 0 0 40,84 0 0 3755991039,21 0 1 1229934080,6 0 0 65535,6 0 0 0' -j REDIRECT --to-port 9053
ip6tables -t nat -I OUTPUT -d fddd:4466:17aa::/48 -p tcp -j REDIRECT --to-ports 9099

The rule was generated using following bits:

• udp[10:1] & 0x80 = 0 — the request should be intercepted and response should not
• udp[12:2] = 1 — there should be single Question in the request
• udp[14:2] = 0 — there should be zero Answers in the request
• udp[16:2] = 0 — there should be zero Authority RRs in the request
• udp[20:1] = 0x10 — QName len should be 16 for old-style onion services
• udp[36:4] & 0x00ffdfdf = 0x054f4e and udp[40:4] & 0xdfdfdfff = 0x494f4e00 ~ \x05onion\x00 + handling for possible 0x20-hack

So those bits combined boil down to:

$ nfbpf_compile RAW '(udp[10:4] & 0x8000ffff = 1) and (udp[14:4] = 0) and (udp[20] = 0x10) and (udp[36:4] & 0x00ffdfdf = 0x054f4e) and (udp[40:4] & 0xdfdfdfff = 0x494f4e00)'
23,48 0 0 0,84 0 0 240,21 0 19 64,48 0 0 9,21 0 17 17,40 0 0 6,69 15 0 8191,177 0 0 0,64 0 0 10,84 0 0 2147549183,21 0 11 1,64 0 0 14,21 0 9 0,80 0 0 20,21 0 7 16,64 0 0 36,84 0 0 16768991,21 0 4 347982,64 0 0 40,84 0 0 3755991039,21 0 1 1229934080,6 0 0 65535,6 0 0 0

I hope it may help to monitor the onion services you have in mind.

[Scapal]

@ageis SOCKS should be supported by blackbox_exporter as it just uses Go http Transport.
Use socks5:// in your proxy url as scheme.

So it all comes down to make one single option available: bypass local DNS resolution.
@brian-brazil Do you want me to submit a PR with such option?

[brian-brazil]

As indicated above, that'd break our v4/v6 features so would not be accepted.

[Scapal]

Using the use_proxy_dns feature would make the ip version selection ignored. That would be logic, in the same way that you don’t use tls options when using unsecured http.
It would be an expected behavior, nothing being broken.

[brian-brazil]

That'd be confusing though, as it's not that endpoint you're testing then but actually the proxy server.

…

On Fri 22 Nov 2019, 17:20 Pascal Fautré, ***@***.***> wrote: Using the use_proxy_dns feature would make the ip version selection ignored. That would be logic, in the same way that you don’t use tls options when using unsecured http. It would be an expected behavior, nothing being broken. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#264?email_source=notifications&email_token=ABWJG5QU7NJW5E4QSBHREZLQVABGTA5CNFSM4EEMTX4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE6EDFQ#issuecomment-557597078>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABWJG5XUQW3TQOET3IEQWBLQVABGTANCNFSM4EEMTX4A> .

[ageis]

@darkk I intend to review your stuff you've posted soon; eventually after my original post I did get a reliable and not-too-convoluted THS-monitoring setup that simply relies upon shoving all requests through Privoxy, and as I recall a few of the roadblocks I had encountered in the beginning have since dissipated; which I think might've been related to the Prometheus team deciding to use miekg/dns.

One of missing bits is that "the DNS servers to be contacted are read from /etc/resolv.conf. But it's possible to redirect DNS queries to *.onion domains to different DNS server with BPF filter at netfilter level.

Thanks for the BPF rule! That is really cool.

[Scapal]

@brian-brazil it is end to end testing, the purpose of blackbox monitoring.
Even without this option, if the test fails, it could be your Ethernet câble, any router between you and the target, a proxy or a reverse proxy along the way.
So I think it doesn't add any confusion.
When I use blackbox monitoring, what I'm trying to know is "does it seems to be accessible for my users" as a whole.

[ageis]

Hey @darkk I'm curious, is it possible to make your netfilter+BPF rule work for version 3 hidden services as well as v2?

[rhatto]

The blackbox exporter is intended to handle a small number of standard of probes. The Tor use case is quite a bit away from that on several axes, so as with anything like that I'd suggest writing a custom exporter rather than trying to overload the blackbox exporter.

Onionprobe fits the use case of Onion Services sites monitoring and was created the after evaluating (and being inspired by) the blackbox exporter.

It comes with built-in Prometheus exporter with many metrics and has a relatively easy setup process.

(Disclosure: I'm the Onionprobe maintainer)