Use LB hostnames as weighted CNAME targets

[maleck13]

Problem
Currently if we hit a LB with a hostname, we resolve the IP addresses and use those as A records. This works but this is not ideal as if the IPs behind that hostname change we wont pick them up.
While we can add a watch for IP changes, this should probably not be the default behaviour as it has some drawbacks

• It could take several minutes to pick up the change and for the DNS change to reach clients.
• Additionally, if the host resolves to multiple IPs we currently give these equal weight but are not actually different clusters.
• Finally having watches running uses up additional resources

Solution

If all the clusters for a given gateway are using LB with hostnames, we should default to structuring the DNS records to be multiple weighted CNAMES

If there is a mix of clusters where a gateway has been deployed with some using an IP as the LB and some using a hostname, we will have to fall back to using just A records as you cannot mix weighted record types (at least in AWS)

Note:

• This is also how we achieved GEO based DNS setup in previous iterations (glbc)
• If a cluster admin were trying to manage this themselves they would have a similar problem if they mixed IPs and hostnames in their cluster LBs. It is not by using MGC that this problem occurs.

Note

I have tested this on AWS but we would want to also figure out what is possible with Azure and Google DNS.

Note: currently our GetAddress method, only works with IPs so this will need to be changed to deal with hosts also

[maleck13]

@mikenairn @roivaz FYI

[philbrookes]

as we cannot mix CNAMEs and A records for a weighted DNS record. When a mix is required, we could generate a (or use a premade e.g. nip.io) CNAME for the IP address we want to include, allowing the mix to function.

Most likely, the ideal approach is to always convert A records to CNAMEs in this manner, so that if a CNAME is required later it can be added.

[maleck13]

this is the DNSRecord resource we want to create

apiVersion: kuadrant.io/v1alpha1
kind: DNSRecord
metadata:
  name: cname-weighted
  namespace: multi-cluster-gateways
spec:
  managedZone:
    name: mgc-dev-mz
  endpoints:
    - dnsName: test.cb.hcpapps.net
      recordTTL: 60
      recordType: CNAME
      setIdentifier: lb1.example.com
      providerSpecific:
       - name: aws/weight
         value: "120"
      labels:
        id: lb1.example.com
      targets:
        - lb1.example.com
    - dnsName: test.cb.hcpapps.net
      recordTTL: 60
      recordType: CNAME
      setIdentifier: lb2.example.com
      providerSpecific:
       - name: aws/weight
         value: "120"
      labels:
        id: lb2.example.com
      targets:
        - lb2.example.com        

[mikenairn]

this is the DNSRecord resource we want to create

This makes sense for hostname only scenarios, but since you can't mix CNAME and A records with the same host (#182 (comment)) do we want to consider creating a CNAME and A record for IP addresses and make that the default in all cases:

spec:
  endpoints:
    - dnsName: test.cb.hcpapps.net
      recordTTL: 60
      recordType: CNAME
      setIdentifier: lb1.example.com
      providerSpecific:
        - name: "aws/weight"
          value: "120"
      labels:
        id: lb1.example.com
      targets:
        - lb1.example.com
    - dnsName: test.cb.hcpapps.net
      recordTTL: 60
      recordType: CNAME
      setIdentifier: lb2.example.com
      providerSpecific:
        - name: "aws/weight"
          value: "120"
      labels:
        id: lb2.example.com
      targets:
        - lb2.example.com
    - dnsName: test.cb.hcpapps.net
      recordTTL: 60
      recordType: CNAME
      setIdentifier: test.127.0.0.1.cb.hcpapps.net
      providerSpecific:
        - name: "aws/weight"
          value: "120"
      labels:
        id: test.127.0.0.1.cb.hcpapps.net
      targets:
        - test.127.0.0.1.cb.hcpapps.net
    - dnsName: test.127.0.0.1.cb.hcpapps.net
      recordTTL: 60
      recordType: A
      setIdentifier: 127.0.0.1
      labels:
        id: 127.0.0.1
      targets:
        - 127.0.0.1

[maleck13]

I see so this would only be needed if there was a mix of hosts and IPs. If it was all IPs we would just use A records, if it was all hosts we would just use CNAME as I outlined. This test.127.0.0.1.cb.hcpapps.net is effectively a made up host to allow us to mix the two.
not ideal, but seems better than hacing to setup watches on hostnames so we can use the IPAddresses

[mikenairn]

I see so this would only be needed if there was a mix of hosts and IPs. If it was all IPs we would just use A records, if it was all hosts we would just use CNAME as I outlined. This test.127.0.0.1.cb.hcpapps.net is effectively a made up host to allow us to mix the two. not ideal, but seems better than hacing to setup watches on hostnames so we can use the IPAddresses

Or just do it all the time for IP addresses, we could always create CNAME records for the managed host instead of doing different things if we happen to have a mix. hostname -> create CNAME, ip -> create dedicated A record and CNAME

[maleck13]

yeah although that doubles the amount of records we need for hosts that already have IPs and doubles the number of look ups as each time a CNAME is encountered it triggers a fresh query

[mikenairn]

yeah although that doubles the amount of records we need for hosts that already have IPs and doubles the number of look ups as each time a CNAME is encountered it triggers a fresh query

Maybe, but the CNAMEs could have a fairly large TTL i would say. As for the number of records, yes you are increasing them, but not by much really. We would essentially be creating an lb CNAME per gateway. If it has multiple IPs, that could just be a single A record with multiple values rather than multiple A records since we don't need to worry about balancing those ips. We probably wouldn't want an ip specific A record, but a generated one:

    - dnsName: test.cb.hcpapps.net
      recordTTL: 300
      recordType: CNAME
      setIdentifier: 12345678987654321.cb.hcpapps.net
      providerSpecific:
        - name: "aws/weight"
          value: "120"
      labels:
        id: 12345678987654321.cb.hcpapps.net
      targets:
        - 12345678987654321.cb.hcpapps.net
    - dnsName: 12345678987654321.cb.hcpapps.net
      recordTTL: 60
      recordType: A
      setIdentifier: 12345678987654321.cb.hcpapps.net
      labels:
        id: 12345678987654321.cb.hcpapps.net
      targets:
        - 127.0.0.1
        - 127.0.0.2
        - 127.0.0.3

In this scenario you'd actually be reducing the record count as currently we create a record per IP.

[maleck13]

Ah I see so we would have a unique host per gateway, as you say a LB host that we put all the IPs under then CNAME to each lb host with a weighted record. So we have 1 A record per gateway and multiple CNAMES to that gateway for each host. I like it. The LB host would really just be a place holder DNSName we would never expect that to be used in a HTTPRoute for example. Will go with this structure for all the things

[maleck13]

So with discussion with @mikenairn, @philbrookes I think your idea is the one we will go with. IE we will set up a DNS name for an IP and CNAME to that and any none IP we will just CNAME to

[maleck13]

This is covered now under the DNSPolicy and DNSRecord proposals. @mikenairn is including this in his work for geo #219