Static / Manual routes management via API

[stevenscg]

I have a use-case where a large number of hostnames (or subdomains) (dozens to a few hundreds) need to be set for a particular consul service. While the number of hostnames is a problem and unwieldy to put into the job file, the hostnames can be added and remove programmatically as customers are provisioned.

I couldn't find any documentation on how to do this via the API, but figured out GET/PUT to /api/manual on the fabio UI port works for this.

I have a request similar to #364 (comment) where more control over adding and removing routes for a given service would be ideal. I looked for a DELETE operation on the API to use when a hostname is added or removed programmatically, but one does not exist.

I also need to provide a route between a host (foo.example.com) and the service (app) when hostnames are added or removed programmatically. Is this supported?

Is this the correct syntax?

route add app foo.example.com/ app

[magiconair]

The idea of the config language approach is that the routing table has a human readable text format where multiple fragments are just concatenated and parsed into a runtime representation. Right now, one fragment is generated from the service tags in consul and the other fragment comes from /fabio/config in the consul KV store. This approach makes parsing simple and robust but doesn't lend itself to API driven partial updates.

Another reason there is no real fabio API for this is that consul already provides an API and a client to manage all this. The idea of the API is from the time when I wanted to support multiple backends besides consul. Given external constraints that doesn't seem to be happening any time soon.

My suggestion is to write a script that generates the routing table and store that under /fabio/config in the consul KV store instead of, or in addition to using service tags. Update the full routing table on every change. This keeps the process simple.

This could be as simple as

routes.py | consul kv put fabio/config

Depending on the size of the routing table there may be some space and/or performance issues. See #343. A single consul KV entry can hold only 512KB and it takes some time to parse lots of them.

The first problem can be addressed by supporting multiple nodes or treat /fabio/config as a directory and parse all subnodes in order. For the parsing speed I need to write a better parser.

The routing syntax is

# host:port is the address app listens on
route add app foo.example.com/ http://host:port

See: https://github.com/fabiolb/fabio/wiki/Routing#config-language

Please let me know if that helps and whether the consul space constraint is an issue for you.

[magiconair]

To find out on which ip addresses and ports app is running you can run a consul or DNS query

dig @localhost -p 8600 srv app.service.consul +short

curl -i localhost:8500/v1/catalog/service/fabio

[magiconair]

Depending on how you implement this you could use the consul watches for this:

https://www.consul.io/docs/agent/watches.html

[stevenscg]

Thanks @magiconair!

The current design certainly makes sense and I like the suggestion of regenerating the routing table with a script in consul. Will try that.

Using multiple consul kv nodes under fabio/config is really interesting and has solid semantics for those of us that work with Consul regularly. I don't see the 512KB as a limiting factor right now.

However, having to watch/manage the dst (http://host:port) is probably the bigger problem for my use case. I think of this more as a "manual route" or a "static route" vs an "override" and a way to provide the same routes as the urlprefix- tags we use when registering consul services.

It sounds like this is not supported today because the dst needs to be a URL and I am trying to use the service name (for dynamic IP:PORT lookup from consul):

route add app foo.example.com/ app

Would this be possible?:

route add app foo.example.com/

It says send requests to foo.example.com/ to the service named app in consul.

[magiconair]

Now I get it. fabio could allow you to provide a template which fills in the host:port so that you just provide the service id and fabio pulls the address itself. That's an interesting case and shouldn't be too hard to add.

[stevenscg]

@magiconair Yup, I think so! I was just trying to replicate what we do in consul service tags with urlprefix to wire up routes.

[magiconair]

Incremental route updates

The issue with incremental route updates is that they are unbounded. So just
keeping a list of all changes and applying them every time won't work.

fabio uses Consul for service discovery and coordination so pushing an
external routing table into a separate Consul KV node seems like the simplest
thing to do.

Consul has a 512KB limit on KV store nodes which translates into 4-5k targets
uncompressed. With gzip this can probably be pushed 2-5x so you'd end up with
10-25k routes of an external routing table. And then you can always have
multiple KV nodes to get around that limit as long as your Consul cluster can
keep up with the write rate.

Depending on the size of your routing table this may also require fixing #343.

Generating route targets

If you generate an external routing table but still want to refer to services
in Consul then fabio can generate them for you. However, the current list of
services and their addresses is not stored and hidden in the registry/consul
package.

I can see several approaches:

1. The registry/consul package caches that list and exposes it.
   This would tie this approach to the consul implementation

2. Services are defined separately as part of the routing table which extends the config language.
   This would decouple the routes from the services. A bit like normalizing an SQL data model.

service svc-a 1.2.3.4:5000 proto=http
service svc-a 1.2.3.5:5000 proto=http
service svc-a 1.2.3.6:5000 proto=http
route add svc-a /foo
route add svc-a /bar

3. fabio expands route add defintions without a target to all known targets.
   This would be shorter but feels like magic and also depends on order.

route add svc-a /foo http://1.2.3.4:5000 
route add svc-a /foo http://1.2.3.5:5000 
route add svc-a /foo http://1.2.3.6:5000 
...
route add svc-a /bar # would expand to three previously defined targets

I think the last option is the last invasive but the second option looks like the cleanest solution.
We could do the last option first as a stepping stone to the second option.

[stevenscg]

@magiconair #417 looks good to get around the consul 512KB limitation.

Option 2 above ("Services are defined separately as part of the routing table") looks interesting.

This would allow a service to be targeted for routing even if it was not in the consul service catalog, right?

If svc-a is already registered in consul, could it be defined like this?

service svc-a http://${ADDR}:${PORT} proto=http
or
service svc-a :${PORT} proto=tcp

Would "tags" mentioned as part of route add be included as part of the service definition then?

service svc-a http://${ADDR}:${PORT} proto=http tags="t1"

Is some kind of a marker needed for the parser to target?

service svc-a consul:http://${ADDR}:${PORT} proto=http
or
service svc-a consul::${PORT} proto=tcp

Option 1 above (registry/consul) seems really interesting even in light of tighter coupling to consul. This could be a competitive and usability advantage (vs something like nginx+, etc.) if I understand what you are proposing.

Stepping back for just a minute...

We have access to really good tooling (git2consul, etc) these days for using consul KV as a configuration store backed by git repositories and have already worked out handling of multiple-regions and multiple environments.

If Fabio gave us the ability to define all routes in consul and also use consul services by name, I would try to use it in lieu of the urlprefix tags.

Here is a real-world example of an api service block in a nomad job that we use within 2 different regions and 2 environments:

service {
    name = "api"
    port = "http"
    tags = [
        "http",
        "urlprefix-api.example.com/",
        "urlprefix-api.us1.example.com/",
        "urlprefix-api.eu1.example.com/",
        "urlprefix-api.example-dev.com/",
        "urlprefix-api.us1.example-dev.com/",
    ]
}

I'm not sure how other users are handling this. Maybe they are templating the job file on submit to nomad, or outside of nomad, have some business logic that provides the appropriate routes as tags for the region and environment. I'm just registering everything for convenience, but it adds weight to the routing table and feels a bit clunky. I also might want the developer of the service to not have to worry about the routing if another team is handling it.

As we discussed earlier in this issue, the urlprefix routes shown are just the static ones that are always present. Subdomain and CNAME routes can be added, changed, or removed for one of our applications dynamically and that's where idea of having Fabio route to healthy members of a service by service name comes in.

Ideally, these route commands (all stored in fabio/config/api or similar) could replace the urlprefix tags from the example and handle the subdomain/CNAME routes:

route add svc-a api.example.com/ tags "http"
route add svc-a api.us1.example.com/ tags "http"
route add svc-a subdomain.example.com/ tags "http"

These route commands omit the <dst> parameter, which is currently required by the parser.

As I understand it, the remaining piece is how to use consul service entries when <dst> is not provided, or as you noted, allow some kind of templating that indicates the service host and port should be obtained from consul.

I prefer the lack of a <dst> parameter to indicate a dynamic lookup, but the templating version could look like:

route add svc-a api.example.com/ http://${ADDR}:${PORT} tags "http"
route add svc-a api.us1.example.com/ http://${ADDR}:${PORT} tags "http"
route add svc-a subdomain.example.com/ http://${ADDR}:${PORT} tags "http"

[magiconair]

The goal of the config language is to provide a textual representation of the routing table so that humans and machines can read it. Otherwise, manual changes would require a different syntax. The urlprefix- tag is already a different representation of the same route add command with a slightly different syntax.

urlprefix-/foo strip=/foo

becomes

route add svc-a /foo http://1.2.3.4:5000 opts "strip=/foo"

When I've developed fabio, nomad didn't really exist and we've deployed self-registering services. The idea was that services know which routes they can handle and by deploying them you make them routable. This way you never have to store a routing table in a separate system (e.g. Puppet) and coordinate deployment of the routing table with the deployment of the service. (routing table before services or after? How to handle roll-back?) You can just forget about this part altogether.

However, different setups require different solutions and if you want to decouple service discovery from defining the routes then having the config language being able to express that would help.

This is why I think option 2 is the cleanest solution since it decouples the two concepts which also means that you can update the service list without knowing about the routing and vice versa.

urlprefix-/foo strip=/foo

becomes

service add svc-a 1.2.3.4:5000
route add svc-a /foo opts "strip=/foo"

Now you can easily integrate different service discovery mechanisms or update the routing or do both. However, when you update the routing manually you have to coordinate this again for every deployment.

I don't think that there would need to be marker to the service name itself but it may be helpful to indicate the origin of the service definition, e.g.

service add svc-a 1.2.3.4:5000 src=consul

[magiconair]

Ah, I think I've closed that one prematurely since this refers to multiple issues. #417 only addresses the 512kb limit. @stevenscg shall we pick the discussion up again if this is still relevant?