Blocky behind Traefik and woes with TLS #576

adyanth · 2022-06-28T20:09:24Z

adyanth
Jun 28, 2022

I am in the process of moving from Pi-Hole to Blocky and let me first say, I am absolutely loving the way this has been implemented, especially the conditional rewrite and mapping, which made me save 4 deployments (one external-dns, bind9 and pihole + cloudflare) which made for a messy setup before <3

I am setting this up in Kubernetes (a k3s cluster). I was able to get the DNS TCP and UDP ports working just fine with a Service LoadBalancer. Now coming to the confusing part of setting up DoH and DoT. From what I understood, I do not need to configure httpsPort and tlsPort if I am putting Traefik in front of it. (I will try by using blocky to serve the cert and see if my problems vanish later)

Issue 1

For DoH, I believe it should be very straightforward as I put Traefik (my ingress controller) in front of the httpPort (in my case, 8080). This makes the API work in both Prometheus and in Grafana for enabling/disabling blocking. But when using DoH for example as shown below, it does not work. Not sure why since a POST with JSON works fine. Please let me know if you see anything wrong here.

Curl logs from DoH

adyanth@ubuntu-nuc:~$ curl -X POST https://blocky.adyanth.lan/api/query -d '{"query": "google.com", "type": "A"}'
{"reason":"RESOLVED (https://1.1.1.1/dns-query)","responseType":"RESOLVED","response":"A (172.217.166.46)","returnCode":"NOERROR"

adyanth@ubuntu-nuc:~$ curl --doh-url https://blocky.adyanth.lan/api/query https://google.com -v
* Found bundle for host blocky.adyanth.lan: 0x5592598eef50 [serially]
* Server doesn't support multiplex (yet)
*   Trying 10.10.100.150:443...
* TCP_NODELAY set
* Hostname 'blocky.adyanth.lan' was found in DNS cache
*   Trying 10.10.100.150:443...
* TCP_NODELAY set
* Connected to blocky.adyanth.lan (10.10.100.150) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* Connected to blocky.adyanth.lan (10.10.100.150) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Jun 28 18:25:06 2022 GMT
*  expire date: Sep 26 18:25:06 2022 GMT
*  subjectAltName: host "blocky.adyanth.lan" matched cert's "blocky.adyanth.lan"
*  issuer: C=IN; ST=Karnataka; O=Adyanth LAN; OU=Home; CN=cert-manager Intermediate CA; [email protected]
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5592598e3800)
> POST /api/query HTTP/2
Host: blocky.adyanth.lan
accept: */*
content-type: application/dns-message
content-length: 28

* We are completely uploaded and fine
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: [NONE]
*  start date: Jun 28 18:25:06 2022 GMT
*  expire date: Sep 26 18:25:06 2022 GMT
*  subjectAltName: host "blocky.adyanth.lan" matched cert's "blocky.adyanth.lan"
*  issuer: C=IN; ST=Karnataka; O=Adyanth LAN; OU=Home; CN=cert-manager Intermediate CA; [email protected]
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x5592598e8fa0)
> POST /api/query HTTP/2
Host: blocky.adyanth.lan
accept: */*
content-type: application/dns-message
content-length: 28

* We are completely uploaded and fine
< HTTP/2 500 
< content-type: text/plain; charset=utf-8
< date: Tue, 28 Jun 2022 19:56:34 GMT
< vary: Origin
< x-content-type-options: nosniff
< content-length: 56
< 
* Connection #0 to host blocky.adyanth.lan left intact
* a DOH request is completed, 1 to go
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 500 
< content-type: text/plain; charset=utf-8
< date: Tue, 28 Jun 2022 19:56:34 GMT
< vary: Origin
< x-content-type-options: nosniff
< content-length: 56
< 
* Connection #1 to host blocky.adyanth.lan left intact
* a DOH request is completed, 0 to go
* DOH: Bad ID type A for google.com
* DOH: Bad ID type AAAA for google.com
* Closing connection 0
curl: (6) Couldn't resolve host name

Issue 2

Next, I tried DoT. I went with having Traefik terminate TLS and forward to 53 on blocky using IngressRouteTCP. I know there are issues both on Traefik for serving the certificate when TLS is requested without SNI and Android (especially since I am using a custom CA, not a Let's Encrypt certificate), so I guess nothing much can be done. At least happy to see this much working!

DoT kdig logs

adyanth@ubuntu-nuc:~$ kdig -d @10.10.100.150 +tls-ca google.com
;; DEBUG: Querying for owner(google.com.), class(1), type(1), server(10.10.100.150), port(853), protocol(TCP)
;; DEBUG: TLS, imported 131 system certificates
;; DEBUG: TLS, received certificate hierarchy:
;; DEBUG:  #1, CN=cluster.k8s.adyanth.site
;; DEBUG:      SHA-256 PIN: f81Ojn+PwspTNk5uo2G6J/JAKZN0XN/jJEdWmqVqtdk=
;; DEBUG:  #2, C=US,O=Let's Encrypt,CN=R3
;; DEBUG:      SHA-256 PIN: jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0=
;; DEBUG:  #3, C=US,O=Internet Security Research Group,CN=ISRG Root X1
;; DEBUG:      SHA-256 PIN: C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=
;; DEBUG: TLS, skipping certificate PIN check
;; DEBUG: TLS, The certificate is NOT trusted. The name in the certificate does not match the expected. 
;; WARNING: TLS, handshake failed (Error in the certificate.)
;; ERROR: failed to query server 10.10.100.150@853(TCP)

adyanth@ubuntu-nuc:~$ kdig -d @10.10.100.150 +tls-ca +tls-host=blocky.adyanth.site google.com
;; DEBUG: Querying for owner(google.com.), class(1), type(1), server(10.10.100.150), port(853), protocol(TCP)
;; DEBUG: TLS, imported 131 system certificates
;; DEBUG: TLS, received certificate hierarchy:
;; DEBUG:  #1, CN=cluster.k8s.adyanth.site
;; DEBUG:      SHA-256 PIN: f81Ojn+PwspTNk5uo2G6J/JAKZN0XN/jJEdWmqVqtdk=
;; DEBUG:  #2, C=US,O=Let's Encrypt,CN=R3
;; DEBUG:      SHA-256 PIN: jQJTbIh0grw0/1TkHSumWb+Fs0Ggogr621gT3PvPKG0=
;; DEBUG:  #3, C=US,O=Internet Security Research Group,CN=ISRG Root X1
;; DEBUG:      SHA-256 PIN: C5+lpZ7tcVwmwQIMcRtPbsQtWLABXhQzejna0wHFr8M=
;; DEBUG: TLS, skipping certificate PIN check
;; DEBUG: TLS, The certificate is trusted. 
;; TLS session (TLS1.3)-(ECDHE-X25519)-(RSA-PSS-RSAE-SHA256)-(AES-128-GCM)
;; ->>HEADER<<- opcode: QUERY; status: NOERROR; id: 3458
;; Flags: qr rd ra; QUERY: 1; ANSWER: 1; AUTHORITY: 0; ADDITIONAL: 1

;; EDNS PSEUDOSECTION:
;; Version: 0; flags: ; UDP size: 1232 B; ext-rcode: NOERROR
;; PADDING: 409 B

;; QUESTION SECTION:
;; google.com.         		IN	A

;; ANSWER SECTION:
google.com.         	249	IN	A	142.250.71.46

;; Received 468 B
;; Time 2022-06-29 01:32:04 IST
;; From 10.10.100.150@853(TCP) in 20.7 ms

My config snippet

    port: 53
    tlsPort: 853
    httpPort: 8080
    httpsPort: 8443
    minTlsServeVersion: 1.3
    #certFile: server.crt
    #keyFile: server.key
    bootstrapDns: tcp+udp:1.1.1.1

Answered by 0xERR0R

Jun 29, 2022

Regarding HTTPS: Yes, if you use traefik, it is the preferred way to use traefik as reverse proxy and do HTTPS. Blocky should only serve traffic on HTTP port, no need to configure https port or any certificate. For testing of DoH, I would recommend https://github.com/ogham/dog. AFAIK is DoH with curl sometimes "tricky".

Regarding DoT with traefik: You are using the IP address for DNS lookup. Does it work if you are using "blocky.adyanth.site"?

View full answer

0xERR0R · 2022-06-29T07:35:40Z

0xERR0R
Jun 29, 2022
Maintainer

Regarding HTTPS: Yes, if you use traefik, it is the preferred way to use traefik as reverse proxy and do HTTPS. Blocky should only serve traffic on HTTP port, no need to configure https port or any certificate. For testing of DoH, I would recommend https://github.com/ogham/dog. AFAIK is DoH with curl sometimes "tricky".

Regarding DoT with traefik: You are using the IP address for DNS lookup. Does it work if you are using "blocky.adyanth.site"?

11 replies

0xERR0R Jun 29, 2022
Maintainer

does it work with dog?

adyanth Jun 29, 2022
Author

I would need to do that some other way since I am not able to run it on my system. It complains of an incorrect libc version.

dog: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by dog)

Will do that later this evening.

adyanth Jun 29, 2022
Author

This is weird, I do not know what is wrong. When I run with dog, I see this in the debug logs:

[2022-06-29 17:09:45] ERROR can't read request: invalid character '¼' looking for beginning of value
[2022-06-29 17:10:33] ERROR can't read request: invalid character '\x13' looking for beginning of value
[2022-06-29 17:10:57] ERROR can't read request: invalid character '|' looking for beginning of value

C:\Users\adyan\Desktop>dog.exe google.com -H @https://cloudflare-dns.com/dns-query
A google.com. 59s   172.217.166.46

C:\Users\adyan\Desktop>wsl curl -X POST https://blocky.adyanth.site/api/query -d '{"query": "google.com", "type": "A"}'
{"reason":"CACHED","responseType":"CACHED","response":"A (216.58.196.78)","returnCode":"NOERROR"}

C:\Users\adyan\Desktop>dog.exe google.com -H @https://blocky.adyanth.site/api/query
Error [http]: Nameserver returned HTTP 500 (Internal Server Error)

C:\Users\adyan\Desktop>dog.exe google.com -H @https://blocky.adyanth.site/api/query
Error [http]: Nameserver returned HTTP 500 (Internal Server Error)

C:\Users\adyan\Desktop>dog.exe google.com -H @https://blocky.adyanth.site/api/query
Error [http]: Nameserver returned HTTP 500 (Internal Server Error)

C:\Users\adyan\Desktop>dog.exe google.com -H @https://blocky.adyanth.site/api/query
Error [http]: Nameserver returned HTTP 500 (Internal Server Error)

0xERR0R Jun 29, 2022
Maintainer

You are using wrong url, please try "https://blocky.adyanth.site/dns-query"

https://blocky.adyanth.site/api/query ist the REST api from blocky (https://0xerr0r.github.io/blocky/swagger.html)

adyanth Jul 1, 2022
Author

Looks like my reply did not come through.
Yes, that was on me for not RTFM :)

DoH works fine now, with no issues. But for DoT, I can see that Traefik receives the query, and logs an entry serving the default LE certificate. But, it never reaches the blocky pod on port 53 even with debug logging.

Here is my Traefik IngressRouteTCP spec, where the dot entrypoint listens on port 853.

spec:
  entryPoints:
  - dot
  routes:
  - match: HostSNI(`*`)
    services:
    - name: blocky
      port: 53
      terminationDelay: 400
      weight: 10
  tls:
    passthrough: false

0xERR0R · 2022-07-01T08:06:03Z

0xERR0R
Jul 1, 2022
Maintainer

hm... My k8s knowledge is very limited. I tried to setup traefik with DoT and DoH and blocky in the past, but only as docker container (without k8s), it worked (something like this https://www.reddit.com/r/pihole/comments/fy8zmu/pihole_traefik_dnsovertls/).

1 reply

adyanth Jul 1, 2022
Author

Cool, will check it out, I had never been able to get that correct even in docker + traefik + Pi-Hole either ;) Thanks for your help, the DoH was a silly one haha

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Blocky behind Traefik and woes with TLS #576

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 12 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Blocky behind Traefik and woes with TLS #576

adyanth Jun 28, 2022

Issue 1

Issue 2

Replies: 2 comments · 12 replies

0xERR0R Jun 29, 2022 Maintainer

0xERR0R Jun 29, 2022 Maintainer

adyanth Jun 29, 2022 Author

adyanth Jun 29, 2022 Author

0xERR0R Jun 29, 2022 Maintainer

adyanth Jul 1, 2022 Author

0xERR0R Jul 1, 2022 Maintainer

adyanth Jul 1, 2022 Author

adyanth
Jun 28, 2022

Replies: 2 comments 12 replies

0xERR0R
Jun 29, 2022
Maintainer

0xERR0R Jun 29, 2022
Maintainer

adyanth Jun 29, 2022
Author

adyanth Jun 29, 2022
Author

0xERR0R Jun 29, 2022
Maintainer

adyanth Jul 1, 2022
Author

0xERR0R
Jul 1, 2022
Maintainer

adyanth Jul 1, 2022
Author