Add support for IPv6

[phillip-stephens]

Description

Adds support for IPv6 name server lookups (nameservers with IPv6 addresses, AAAA lookups were already supported) and binding to IPv6 local addresses.

IP mode is selected with the following criteria:

1. If user provides --4 or --6, we'll use only IPv4 or only IPv6. This flags cannot be combined
2. If user provides nameservers, we'll infer IPv4/v6 support thru nameservers
3. Infer IPv4/6 support thru what nameservers are listed in /etc/resolv.conf

Adds a new --prefer-ipv4-iteration and --prefer-ipv6-iteration options that specifically affect iteration when in both IPv4/6. When we're iterating thru the Authorities and we prefer IPv4, we'll prefer to request A records if they exist, but AAAA are used as a backup. The reverse is true for --prefer-ipv6-iteration.

IPv6 nameservers are treated the same as IPv4 in the way that they're added to the list of external/root nameservers. Each resolver has one list of these and IPv4/IPv6 addresses are mixed. Whenever a random Nameserver is pulled for a lookup, we choose the relevant ConnectionInfo object (a new wrapper around a localAddr, dns.Conn, tcp/udp connection), either IPv4 or IPv6.

A couple other notes:

• loopback/link-local IPv6 addresses are not supported
• If the user selects IPv6Only/--6 and we can't bind to an IPv6 local address, we error. Same for IPv4Only
• We cache full responses, we don't try to filter out IPv6 Answers/Additionals if we're in IPv4Only mode, since that could result in responses not including all records related to a domain query
• During a cached lookup, it's quite likely a cache entry includes info for both A/AAAA. If we're in IPv4Only and we hit a AAAA record, we'll just skip it since we can't follow that authority. Same is true for IPv6Only. The cache will include everything, but when we read from the cache we'll just avoid following records we wouldn't be able to resolve.

Testing

Testing in any sort of automated way is difficult. Github actions don't seem to support IPv6 and our lab VMs don't yet support IPv6 either. Fortunately, my home connection does have IPv6 so I've included the testing I've done below. Additionally, I've added a make target make ipv6-tests that run a new python file ipv6_tests.py. These cannot be included with our main suite because anyone that runs these on a host w/o IPv6 support will see tests fail. So presently these are a "test manual on your machine that hopefully has IPv6" sort of deal.

Test Cases

• Normal Usage unaffected (ZDNS detects host's IPv6 support thru OS configured nameservers)

$ echo "cloudflare.com" | ./zdns A --verbosity=5  
INFO[0000] No iteration IP preference specified, defaulting to IPv4 preferred. See --prefer-ipv4-iteration and --prefer-ipv6-iteration for more info 
INFO[0000] using local addresses: [192.168.1.243 2603:6013:9d00:3302:b5f3:16eb:90b1:1c5] 
INFO[0000] for non-iterative lookups, using external nameservers: 192.168.1.1:53, [2603:6013:9d00:3302::1]:53 
INFO[0000] for iterative lookups, using nameservers: 192.168.1.1:53, [2603:6013:9d00:3302::1]:53 
INFO[0000] no name server provided for external lookup, using  random external name server: 192.168.1.1:53 
...
{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":4096,"version":0}],"answers":[{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":209,"type":"A"},{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":209,"type":"A"}],"protocol":"udp","resolver":"192.168.1.1:53"},"duration":0.031517708,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-14T12:41:44-04:00"}

• IPv6-Only Mode

$ echo "cloudflare.com" | ./zdns A --verbosity=5  --6
INFO[0000] using local addresses: [2603:6013:9d00:3302:b5f3:16eb:90b1:1c5] 
INFO[0000] for non-iterative lookups, using external nameservers: [2603:6013:9d00:3302::1]:53 
INFO[0000] for iterative lookups, using nameservers: [2603:6013:9d00:3302::1]:53 
INFO[0000] no name server provided for external lookup, using  random external name server: [2603:6013:9d00:3302::1]:53 
...
{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":4096,"version":0}],"answers":[{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":111,"type":"A"},{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":111,"type":"A"}],"protocol":"udp","resolver":"[2603:6013:9d00:3302::1]:53"},"duration":0.012365042,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-14T12:43:21-04:00"}

• Resolve a domain using Google's IPv6 DNS endpoint ([2001:4860:4860::8888]:53)

echo "cloudflare.com" | ./zdns A --verbosity=5 --name-servers="2001:4860:4860::8888" 
INFO[0000] using local addresses: [2603:6013:9d00:3302:b5f3:16eb:90b1:1c5] 
INFO[0000] for non-iterative lookups, using external nameservers: [2001:4860:4860::8888]:53 
INFO[0000] for iterative lookups, using nameservers: [2001:4860:4860::8888]:53 
INFO[0000] no name server provided for external lookup, using  random external name server: [2001:4860:4860::8888]:53 
...
{"data":{"additionals":[{"flags":"","type":"EDNS0","udpsize":512,"version":0}],"answers":[{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"},{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"}],"protocol":"udp","resolver":"[2001:4860:4860::8888]:53"},"duration":0.0573195,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-14T12:43:59-04:00"}

• IPv6 Mode Iteration

$ echo "cloudflare.com" | ./zdns A --verbosity=4 --6 --iterative
WARN[0000] loopback external IPv6 nameservers are not supported: [::1]:53, using ZDNS defaults: [[2001:4860:4860::8888]:53 [2001:4860:4860::8844]:53 [2606:4700:4700::1111]:53 [2606:4700:4700::1001]:53] 
{"data":{"answers":[{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"},{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"}],"protocol":"udp","resolver":"[2400:cb00:2049:1::a29f:21]:53"},"duration":0.109435208,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-07T16:38:07-04:00"}

• --prefer-ipv4-iteration (Notice in the DEBUG logs how we're following IPv4)

$ echo "cloudflare.com" | ./zdns A --verbosity=5 --iterative --prefer-ipv4-iteration
INFO[0000] using local addresses: [192.168.1.243 2603:6013:9d00:3302:f9d5:1f65:a415:75a7] 
INFO[0000] for non-iterative lookups, using nameservers: 192.168.1.1:53, [2603:6013:9d00:3302::1]:53 
...
DEBU[0000] DEPTH 02:        [-> Authority found, iterating] 
DEBU[0000] DEPTH 02:        [Trying Authority:  {172800 NS 2 IN 1 com k.gtld-servers.net.}] 
DEBU[0000] DEPTH 02:        [Output from extract authorities:  192.52.178.30:53] 
DEBU[0000] DEPTH 02:        [iterative lookup for  cloudflare.com  ( 1 ) against  192.52.178.30:53  layer  com] 
...
{"data":{"answers":[{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"},{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"}],"protocol":"udp","resolver":"162.159.0.33:53"},"duration":0.255885084,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-02T11:35:35-04:00"}

• --prefer-ipv6-iteration

echo "cloudflare.com" | ./zdns A --verbosity=5 --iterative --prefer-ipv6-iteration
INFO[0000] using local addresses: [192.168.1.243 2603:6013:9d00:3302:f9d5:1f65:a415:75a7] 
INFO[0000] for non-iterative lookups, using nameservers: 192.168.1.1:53, [2603:6013:9d00:3302::1]:53 
...
DEBU[0000] DEPTH 03:            [-> Authority found, iterating] 
DEBU[0000] DEPTH 03:            [Trying Authority:  {172800 NS 2 IN 1 cloudflare.com ns3.cloudflare.com.}] 
DEBU[0000] DEPTH 03:            [Output from extract authorities:  [2400:cb00:2049:1::a29f:21]:53] 
DEBU[0000] DEPTH 03:            [iterative lookup for  cloudflare.com  ( 1 ) against  [2400:cb00:2049:1::a29f:21]:53  nswers":[{"answer":"104.16.132.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"},{"answer":"104.16.133.229","class":"IN","name":"cloudflare.com","ttl":300,"type":"A"}],"protocol":"udp","resolver":"[2400:cb00:2049:1::a29f:21]:53"},"duration":0.132352625,"name":"cloudflare.com","status":"NOERROR","timestamp":"2024-08-02T11:38:42-04:00"}

Performance

With changes of this magnitude, I wanted to be sure our accuracy/error rate wasn't decreasing. Compared to main, we're seeing similar IPv4 performance:

main - IPv4 - make benchmark

$ make benchmark 
...
Benchmark took:                                                       29.26s
Min resolution time:                                                 21.70µs
Max resolution time:                                                  15.00s
Average resolution time:                                            258.67ms

Domains resolved successfully:                                     6913/7000
Domains that timed out:                                                   12
Domains that failed:                                                      75

Phillip/ipv4 - IPv4 - make benchmark

Benchmark took:                                                       28.75s
Min resolution time:                                                 22.72µs
Max resolution time:                                                  15.00s
Average resolution time:                                            243.78ms

Domains resolved successfully:                                     6914/7000
Domains that timed out:                                                   11
Domains that failed:                                                      75

IPv6

Ofc we can't measure IPv6 performance compared to main since this is a new feature, but you can compare the runtimes to the IPv4 runs above. These were performed on a Digital Ocean IPv6 VM with 1 core and 1 GB RAM. ./benchmark/main.go was modified to run zdns with --6 to use IPv6.

make benchmark, IPv6-only, Phillip/ipv6 branch

Benchmark took:                                                      115.33s
Min resolution time:                                                 40.84µs
Max resolution time:                                                  15.01s
Average resolution time:                                               1.51s

Ten longest resolutions:
        www.dhl.com:                                                  15.01s
        ph.search.yahoo.com:                                          15.01s
        zhidao.baidu.com:                                             15.01s
        cav.receita.fazenda.gov.br:                                   15.01s
        www.teerresults.com:                                          15.00s
        udyamregistration.gov.in:                                     15.00s
        newmanager.cityheaven.net:                                    15.00s
        www.lecturas.com:                                             15.00s
        seller.tokopedia.com:                                         15.00s
        www.nj.com:                                                   14.36s

Domains resolved successfully:                                     5656/7000
Domains that timed out:                                                  133
Domains that failed:                                                    1209

Most failed domains (1080/1209) failed with NONEEDEDGLUE, meaning it's likely their nameservers weren't capable of IPv6.

For the timeouts, I selected a handful and checked them manually and they were again not successful:

echo "laodong.vn\nwww.solarweb.com\nwww.starfall.com\nportal.auone.jp\ntwitter.com\nwww.wineandfoodtour.it" | ./zdns A --iterative --6
{"data":{"protocol":"","resolver":""},"duration":7.237384887,"name":"www.starfall.com","status":"SERVFAIL","timestamp":"2024-08-14T15:40:57Z"}
{"data":{"protocol":"udp","resolver":"[2001:500:14:6115:ad::1]:53"},"duration":15.000967412,"name":"laodong.vn","status":"TIMEOUT","timestamp":"2024-08-14T15:41:05Z"}
{"data":{"protocol":"","resolver":""},"duration":15.000846996,"name":"twitter.com","status":"TIMEOUT","timestamp":"2024-08-14T15:41:05Z"}
{"data":{"protocol":"","resolver":""},"duration":15.001229404,"name":"portal.auone.jp","status":"NONEEDEDGLUE","timestamp":"2024-08-14T15:41:05Z"}
{"data":{"protocol":"","resolver":""},"duration":15.000935505,"name":"www.wineandfoodtour.it","status":"TIMEOUT","timestamp":"2024-08-14T15:41:05Z"}
{"data":{"protocol":"","resolver":""},"duration":15.001255172,"name":"www.solarweb.com","status":"TIMEOUT","timestamp":"2024-08-14T15:41:05Z"}

[zakird]

I think that this looks good, though it's a bit hard for me to tell if we've caught everything.