One second delay to resolve DNS query on localhost?

[shreeve]

If I run the following code,

#!/usr/bin/env ruby

require 'async/dns'

$host = '127.0.0.1'
$port = 2346

class AutoServer < Async::DNS::Server
  def process(who, rsc, txn)
    txn.respond! $host
  end
end

puts "autoserver listening on #{$host}:#{$port}"

auto = AutoServer.new([[:udp, $host, $port]])
auto.run

Each response seems to be throttled at 1 second minimum. For example:

$ time dig @localhost -p 2346 google.com > /dev/null

real	0m1.018s
user	0m0.006s
sys	0m0.007s

$ time dig @localhost -p 2346 google.com > /dev/null

real	0m1.021s
user	0m0.006s
sys	0m0.007s

$ time dig @localhost -p 2346 google.com > /dev/null

real	0m1.018s
user	0m0.006s
sys	0m0.007s

The 1.018s, 1.021s, and 1.018s are so close to 1.000s that they imply there is a 1 second timeout somewhere.

Any ideas?

[ioquatix]

Try

time dig @127.0.0.1 -p 2346 google.com

[ioquatix]

Try

time dig @::1 -p 2346 google.com

[ioquatix]

You need something like

auto = AutoServer.new([
  [:udp, '127.0.0.1', $port],
  [:udp, '::1', $port]
])

Honestly, RubyDNS / async-dns is one of the oldest parts of this project and should be updated to fit with how endpoints work now.

[shreeve]

Using @127.0.0.1 and @::1 fixed the issue completely!

What is a quick summary of the reason why?

[ioquatix]

When you run the server on IPv4 only, but you try to use dig localhost, it appears to me that localhost is resolving to IPv6. It tries to connect, times out, and the tries IPv4. This connects, services the request and everyone is at peace with the universe.

[ioquatix]

Let me know if you need more details. Happy to explain in explicit detail.

[shreeve]

If we hook this into a resolver, can this little bit of code be used instead of having to run dnsmasq?

[ioquatix]

Yes, people have already done it, but it's something worth exploring if it interests you.

https://rubygems.org/gems/rubydns/reverse_dependencies

e.g. vagrant-dns

[ioquatix]

https://github.com/fnordfish/matchd

https://github.com/BerlinVagrant/vagrant-dns

https://github.com/vagrant-landrush/landrush

[shreeve]

On macos we can create a resolver for a tld that is mapped to this dns server, like this:

# macos: create a resolver for a TLD
tld=test
sudo mkdir -p /etc/resolver
sudo bash -c "echo -e 'nameserver 127.0.0.1\nport 2346' > /etc/resolver/${tld}"

This resolver would then "answer for" any requests being made to our given TLD (which is test above). So a host such as foo.test would be directed to the DNS server we are running on 127.0.0.1:2346 and it would resolve to 127.0.0.1.

By doing things this way, we have a super simple way to resolve wildcard hostnames and always return our 127.0.0.1 address.

If we combine this with creating a single wildcard SSL certificate for the same domain, we wouldn't need to do all the complex stuff with LetsEncrypt. I've got a way to create a simple wildcard SSL cert and a way to auto-trust it on mac.

This would "close the loop" and allow us to support wilcard https for development, in a very simple way!

[ioquatix]

That makes sense, and it's great to see a summary of how it all fits together.

I would be cautious of putting platform specific stuff into falcon.

That being said, maybe there is room for a tool to do this in development.

The purpose of Lets Encrypt/ACME protocol is for seamless production deployment.

Your enthusiasm makes me happy.

[ioquatix]

For example, this concept might fit with the localhost gem.

https://github.com/socketry/localhost

If we can support both Linux and macOS, that would be ideal. If we put this into that gem, it probably makes the most sense to me.

Essentially:

Generate a certificate for this hostname with this TLD, and make sure the local OS responds to it. It's not directly part of certificate creation, but I imagine it can be useful to expose another interface that works together well.

[shreeve]

I've got something working that should be a great option for "seamless, out of the box" local development with Falcon using HTTP/2 (which requires https and SSL/TLS certificates).

Let's say that we are developing three local apps, foo, bar, and baz. In order to do secure local development with Falcon, we need to have two main things in place:

1. We need to be able to enter a URL in our browser and be pointed to our app, served by Falcon
2. We need to have a valid SSL certificate associated with each app, so that HTTP/2 works

You can handle (1) by manually updating the /etc/hosts file or by installing dnsmasq (or another DNS server) to map a hostname each app to 127.0.0.1 where Falcon is running. It's common to create entries in the hosts file that look like 127.0.0.1 foo.local.dev. This way, you can type foo.local.dev in your browser and it'll get mapped to 127.0.0.1, where Falcon is waiting to handle the request. Entries can then be added to the hosts file like 127.0.0.1 bar.local.dev and 127.0.0.1 baz.local.dev. It's a little tedious, but not that bad. It would be great to be able to use a wildcard like 127.0.0.1 *.local.dev, but wildcards aren't valid in the hosts file. You need to enter a new entry for each app that you want to serve. In order to get around this, dnsmasq (or another DNS server) is frequently used to handle wildcard mappings, so that you don't need to tweak /etc/hosts for each new app.

You can handle (2) by either creating a new self-signed certificate for each app and then importing it into the certificate store or using the ACME protocol with a service like LetsEncrypt to generate a per-site or per-app SSL certificate. These approaches require manual tweaking for each app and installing additional software to run a LetsEncrypt agent and have it periodically update the certificates, since they have relatively short expiration periods.

In short, both areas don't really have good solutions, which can make it sort of painful to use them with HTTP/2 for local development with Falcon.

To solve these two items, we can do the following:

1. Run a DNS server as part of Falcon, so that it will handle the wildcard DNS resolution for us, for free
2. Use a wildcard SSL certificate that works for all of our apps and only needs to be loaded once in our certificate store

For (1), we actually have two things that we need to do. Not only do we need to run a DNS server that will respond with 127.0.0.1 for all hostname lookups, but we need to tell our operating system where to find that DNS server, since we aren't using a "real" TLD (top level domain) like .com or .net. So, the first thing we need to do is setup a "resolver" which tells our operating system where it should find the DNS server to use to translate our foo.local.dev to 127.0.0.1 or any other app, such as bar.local.dev and baz.local.dev. These instructions are for macOS, but a similar approach exists for Linux and Windows:

# macOS: Create a resolver for our base domain
base=local.dev
sudo mkdir -p /etc/resolver
sudo bash -c "echo -e 'nameserver 127.0.0.1\nport 2346' > /etc/resolver/${base}"

If we now try to lookup a host such as anything.local.dev, the operating system will point us to a DNS server running on port 2346 of 127.0.0.1. This is where we want our Falcon-powered DNS server to be waiting with a handy reply that the requested host is also found at address 127.0.0.1.

The DNS server that Falcon can launch for us looks like this:

#!/usr/bin/env ruby

require 'async/dns'

$ipv4 = '127.0.0.1'
$ipv6 = '::1'
$port = 2346

class AutoServer < Async::DNS::Server
  def process(who, rsc, txn)
    txn.respond! $ipv4 # $ipv4
  end
end

puts "autoserver listening on #{$ipv4}:#{$port}"

auto = AutoServer.new([
  [:udp, $ipv4, $port],
  [:tcp, $ipv4, $port],
])
auto.run

With the above resolver and server, then any request such as something.local.dev will be forwarded by the resolver to our DNS server, which responds with the desired result:

ping -o something.local.dev
PING something.local.dev (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.036 ms

--- something.local.dev ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.036/0.036/0.036/0.000 ms

So, with very little effort we are now able to resolve any of our apps' hostnames to point to Falcon. One down, one to go.

For (2), we need a way to generate a wildcard SSL certificate that will work for all of our apps. On macOS, here's how this was accomplished:

# enter our base domain (we'll prepend each app name)
base=local.dev

# create the config file for the wildcard SSL certificate
cat > ${base}.cnf <<-end
[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

[req_distinguished_name]
CN = ${base}

[v3_req]
keyUsage = digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[alt_names]
DNS.1 = ${base}
DNS.2 = *.${base}
end

# from the wildcard SSL config file, generate the key and cert files
openssl req -new -newkey rsa:2048 -days 3650 -nodes -x509 -sha1 \
  -config ${base}.cnf \
  -keyout ${base}.key \
  -out    ${base}.crt

# import and trust our new wildcare SSL certificate
sudo /usr/bin/security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain ${base}.crt

As a side note, these local.dev.crt and local.dev.key files can be shared by anyone using Falcon, wanting to host apps using those domains from their local machine. So, these steps aren't really needed to be taken by all users, we could have these two files are part of the Falcon repo. But anyway...

Now that we have created the key and cert files, we give them to Falcon so we can properly serve secure content over HTTP/2. Here's the server.rb file that launches Falcon and reads in the correct wildcard SSL key and cert files:

#!/usr/bin/env ruby
require "bundler/setup"

require "falcon"
require "async"
require "async/http"

# variables
cfgfile = "config.ru"
pub_url = "https://foo.local.dev:8443"
ssl_crt = "local.dev.crt"
ssl_key = "local.dev.key"
ps_name = "Falcon Server"

Async.logger.level = Logger::INFO

context = OpenSSL::SSL::SSLContext.new

context.add_certificate(
  OpenSSL::X509::Certificate.new(File.read(ssl_crt)),
  OpenSSL::PKey.read(File.read(ssl_key))
)

endpoint = Async::HTTP::URLEndpoint.parse(pub_url, ssl_context: context)

bound_endpoint = Async do
  Async::IO::SharedEndpoint.bound(endpoint)
end.wait

rack_app, options = Rack::Builder.parse_file(cfgfile)

middleware = Falcon::Server.middleware(rack_app)

Async::Container::Forked.new(concurrency: 8, name: ps_name) do |task, instance|
  server = Falcon::Server.new(middleware, bound_endpoint, endpoint.protocol, endpoint.scheme)
  server.run
  task.children.each(&:wait)
end

With this in place, we just launch Falcon by running server.rb. With our DNS server and app server running, we can now fire up a browser and go to app.local.dev to see our running app in all of it's glory over HTTP/2 and secure https/SSL/TLS.

==

NOTE: This is still pretty experimental, the instructions above are only for macOS, they don't really show how to actually serve up multiple apps from within Falcon (although all the hard parts are explained above). In essence, this "pretty much works", but needs some love to make it solid.

Also, for those interested... on the SSL certificates you can create certificates as follows:

# non-wildcard certs work fine for hosts like this:
dev
local.dev
app.local.dev

# these wildcard names are illegal
*
*.dev

# but, this wilcard name works!
*.local.dev

If you notice above, we use the following in the certificate configuration file:

[alt_names]
DNS.1 = ${base}
DNS.2 = *.${base}

This means that our base domain (local.dev) and a wildcard based on it (ie - *.local.dev) are "included" in the certificate. This allows both forms to be used in the same certificate. This approach can be used to add many more hosts and wildcards to the certificate, if desired. Keep in mind that using a wildcard requires at least two non-wildcard prefixes. So, the broadest wildcard is *.domain.tld.

This blurb has turned into a long-winded crazy post, but I wanted to try to at least capture everything here in one place for future reference (or ridicule!).

[shreeve]

Sorry that this is posted to async/dns, just sort of coming as a follow-up to your message above @ioquatix.

[ioquatix]

This all makes sense and is a good proof of concept. I think the logical place for the certificate stuff is in localhost. And it might even make sense to put the DNS server implementation there too. That way, it's agonistic to the web server implementation, which I think is probably a good idea to keep things isolated and layered appropriately. What do you think?

[ioquatix]

@shreeve where did we get to with this?

[ioquatix]

The original issue has been resolved, so closing this.

Feel free to open a discussion if you want to discuss how the other parts of this issue can be moved forward. Note that my recommended approach for local development with TLS is to use FQDNs e.g. mywebsite.localhost and this is supported by the localhost gem.