Choosing a specific NIC when there are multiple NICs with calico and kube-vip

[onedr0p]

Details

This is advanced configuration but I figured it would be worth opening an issue to inform people how to deal with situations where your nodes have multiple active NICs and you want to choose which one Kubernetes uses.

If you nodes have different network interface names, it's advised to make the names consistent across devices. Review this question on AskUbuntu on how to make that happen.

These changes should be made prior to running task configure in the guide.

Calico

By default, attaches to the first available internet interface (https://projectcalico.docs.tigera.io/networking/ip-autodetection).

The way around this could be achieved by setting a interface name regex in the Calico Installation resource.

# ./provision/ansible/playbooks/templates/calico-installation.yaml.j2
---
apiVersion: operator.tigera.io/v1
kind: Installation
metadata:
  name: default
spec:
  calicoNetwork:
    nodeAddressAutodetectionV4:
      interface: "eno.*"
...

kube-vip

By default kube-vip choose the NIC with the default gateway. To assign a specific NIC update the configuration below.

# ./provision/ansible/playbooks/templates/kube-vip-daemonset.yaml.j2
...
          env:
...
            - name: vip_interface
              value: "eno1"
...

# ./tmpl/cluster/kube-vip-daemonset.yaml
...
          env:
...
            - name: vip_interface
              value: "eno1"
...

[brettinternet]

Perhaps we could automate this with Ansible using the CIDR for IP detection? https://projectcalico.docs.tigera.io/networking/ip-autodetection

spec:
  calicoNetwork:
    nodeAddressAutodetectionV4:
      cidrs:
        - "{{ (ansible_default_ipv4.network + '/' + ansible_default_ipv4.netmask) | ipaddr('network/prefix') }}" # 192.168.200.0/24

[onedr0p]

@brettinternet how would that help when a node has multiple active NICs, and for example each nic is part of a different network?

[brettinternet]

Oh, I just mean as a default for hardware with multiple NICs but where only one is used. Otherwise I believe they'll find these Calico errors on those nodes:

[INFO][76] felix/route_table.go 1200: Failed to access interface because it doesn't exist. error=Link not found ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
[INFO][76] felix/route_table.go 1268: Failed to get interface; it's down/gone. error=Link not found ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
[ERROR][76] felix/route_table.go 1035: Failed to get link attributes error=interface not present ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0
[INFO][76] felix/route_table.go 618: Interface missing, will retry if it appears. ifaceName="vxlan.calico" ifaceRegex="^vxlan.calico$" ipVersion=0x4 tableIndex=0

[onedr0p]

@brettinternet this should help certain people so if you want to open a PR to add that, please do otherwise I'll get to it later.

Edit: Done in effd4dd

[mrueg]

@onedr0p Have you this or a similar configuration for cilium as well?

My usecase:
I want to set up two ingresses, one private, one public one. The private ingress should be exposed via tailscale on an internal network, the public one via the default interface to the internet.

[onedr0p]

I have not tried this but hopefully the Cilium docs can have useful info on how to accommodate that.