CCM: Incorrect network interfaces order, when multiple interfaces are used

[kayrus]

/kind bug

What happened:

When k8s compute nodes have multiple network interfaces, CCM constantly change them:

$ kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP     OS-IMAGE                                      KERNEL-VERSION               CONTAINER-RUNTIME
node0012   Ready    <none>   18d   v1.16.3   10.128.0.34   192.1.106.99    Red Hat Enterprise Linux Server 7.7 (Maipo)   3.10.0-1062.1.2.el7.x86_64   docker://18.9.7
node0013   Ready    <none>   18d   v1.16.3   10.1.0.10     192.1.106.76    Red Hat Enterprise Linux Server 7.7 (Maipo)   3.10.0-1062.1.2.el7.x86_64   docker://18.9.7
$ sleep 5m
$ kubectl get nodes -o wide
NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP   EXTERNAL-IP     OS-IMAGE                                      KERNEL-VERSION               CONTAINER-RUNTIME
node0012   Ready    <none>   18d   v1.16.3   10.1.0.34     192.1.106.99    Red Hat Enterprise Linux Server 7.7 (Maipo)   3.10.0-1062.1.2.el7.x86_64   docker://18.9.7
node0013   Ready    <none>   18d   v1.16.3   10.1.0.10     192.1.106.76    Red Hat Enterprise Linux Server 7.7 (Maipo)   3.10.0-1062.1.2.el7.x86_64   docker://18.9.7

This behavior causes issues, when kube-apiserver tries to connect to kubelet for logs or exec command, when 10.128.0.0/24 subnet is not available for kube-apiserver

And loadbalancer pool members change every sync loop:

I1203 19:28:31.845065       1 openstack_loadbalancer.go:1130] Ensured pool 0027f480-d407-44f3-bb5d-7d7c95fcbe2d has member for node0013 at 10.1.0.34
I1203 19:28:31.845271       1 openstack_loadbalancer.go:1135] Deleting obsolete member adfed073-bade-46cf-8b16-3bdfed2c7a62 for pool 0027f480-d407-44f3-bb5d-7d7c95fcbe2d address 10.128.0.34

What you expected to happen:

I expect to see the first interface IP address for all nodes, or at least don't change the IP address orders on every reconciliation loop.

How to reproduce it (as minimally and precisely as possible):

Create VMs with more than one network interfaces.

Anything else we need to know?:

golang/go#27179 golang issue.

Even setting the alpha.kubernetes.io/provided-node-ip node annotation or specifying the --node-ip kubelet parameter doesn't solve the issue.

one liner to get the first interface private ip address on the node

ip -o -4 addr show dev $(find /sys/class/net -type l -not -lname '*virtual*' -printf '%f\n' -quit) | cut -d ' ' -f 7 | cut -f 1 -d '/'

Just for the history, bugs with slightly similar symptoms:

• unexpected providerID format digitalocean/digitalocean-cloud-controller-manager#126
• Kubelet reports secondary InternalIP in AWS with multiple ENIs kubernetes#61921

Environment:

• openstack-cloud-controller-manager version: 1.16
• OS (e.g. from /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Others:

refs:

• Should we remove the alpha.kubernetes.io/provided-node-ip annotation?  kubernetes#84490

[kayrus]

@ramineni @lingxiankong @adisky

I see the following ways to fix the root cause:

• fix in gophercloud with json.RawMessage + ordered JSON map implementations
• introduce a kubelet sidecar (this was possible when there was in-tree cloud provider), which will update the address from the local metadata service or according to local ethernet devices.
• figure out why kube-apiserver ignores alpha.kubernetes.io/provided-node-ip annotation

[kayrus]

According to this code https://github.com/kubernetes/kubernetes/blob/16c2ae4607371f1f948c0a9c884805e699330450/pkg/kubelet/nodestatus/setters.go#L76..L91 kubelet won't set the alpha.kubernetes.io/provided-node-ip annotation, when external cloud provider is used.

When annotation is set, kube-apiserver still gets updates from CCM, and it just verifies whether an annotated private IP is in the status list: https://github.com/kubernetes/kubernetes/blob/16c2ae4607371f1f948c0a9c884805e699330450/pkg/controller/cloud/node_controller.go#L560..L573

Another interesting behavior. When you set --cloud-provider=openstack for kubelet and --cloud-provider=openstack for CCM, they will fight with each other setting the IP addresses. kubelet will have his truth, CCM - his.

It is possible to modify the updateNodeAddress func and prioritize the IP address, specified in the annotation. This will require code change in kubernetes, which is usually not fast.

Modifying gophercloud with the json.RawMessage solution is way faster.

[kayrus]

Found a way to retrieve IP addresses in a proper order, where no maps are involved: https://docs.openstack.org/api-ref/compute/?expanded=list-port-interfaces-detail#port-interfaces-servers-os-interface