Multi-nic node ip must correspond to kubelet api listening address

[mogaika]

/kind bug

What happened:
openstack cloud provider replaces nodes internal ip with openstack node instance addresses, but kubelet api of node can be bounded only to one of these addresses (e.g. for security purposes). Typical scenario: first nic - for deployment, second nic- for k8s network and kubelet --address parameter equals to the second nic ip. In this case cluster can receive wrong address of node kubelet api (because usually selects first nic ip as cluster ip) and features like "kubectl logs" and "kubectl proxy" will not work.

What you expected to happen:
Possibility to declare control-network in cloud config file.
Or declare kubelet listen addresses per node.

How to reproduce it (as minimally and precisely as possible):
Use multiple nics for kubernetes nodes and set kubelet --address parameter to one of them
Check that wrong ip was assigned to internal-ip field in kubectl get nodes -o wide output
Try to kubectl exec to pod that was sheduled to node with wrong ip.

Anything else we need to know?:
from opentack-cloud-provider logs
Error patching node with cloud ip addresses = [failed to patch status "{\"status\":{\"$setElementOrder/addresses\":[{\"type\":\"InternalIP\"},{\"type\":\"ExternalIP\"},{\"type\":\"InternalIP\"}],\"addresses\":[{\"address\":\"172.16.10.9\"type\":\"InternalIP\"},{\"type\":\"ExternalIP\"},{\"type\":\"InternalIP\"}],\"addresses\":[{\"address\":\"172.16.10.95\",\"type\":\"InternalIP\"},{\"address\":\"192.168.10.102\",\"type\":\"InternalIP\"}]}}" for node "cmp1.****": Node "cmp1.****" is invalid: status.addresses[1]: Duplicate value: core.NodeAddress{Type:"InternalIP", Address:"192.168.10.102"}]

Environment:

• openstack-cloud-controller-manager version:
• OS (e.g. from /etc/os-release): Ubuntu 16.04.1 LTS (Xenial Xerus)
• Kernel (e.g. uname -a): Linux ctl03 4.4.0-36-generic RBAC info for external cloud provider / CCM #55-Ubuntu SMP Thu Aug 11 18:01:55 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: saltstack
• Node networks:
  nic1 - 192.168.10.0/24
  nic2 - 172.16.10.0/24

• floating ip as external network - 10.13.0.0/16

• Example of status.addresses of node:

    addresses:
    - address: 192.168.10.94
      type: InternalIP
    - address: 10.13.250.32
      type: ExternalIP
    - address: 172.16.10.112
      type: InternalIP

• Output of kubectl get nodes -o wide:

NAME         STATUS   ROLES    AGE    VERSION                    INTERNAL-IP      EXTERNAL-IP    OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
cmp0.****    Ready    node     12h    v1.12.3-2+bc2e990ce4e046   192.168.10.98    10.13.250.12   Ubuntu 16.04.1 LTS   4.4.0-36-generic   docker://1.13.1
cmp1.****    Ready    node     14h    v1.12.3-2+bc2e990ce4e046   172.16.10.95     10.13.250.26   Ubuntu 16.04.1 LTS   4.4.0-36-generic   docker://1.13.1
ctl01.****   Ready    master   4d3h   v1.12.3-2+bc2e990ce4e046   192.168.10.104   10.13.250.25   Ubuntu 16.04.1 LTS   4.4.0-36-generic   docker://1.13.1
ctl02.****   Ready    master   14h    v1.12.3-2+bc2e990ce4e046   192.168.10.92    10.13.250.19   Ubuntu 16.04.1 LTS   4.4.0-36-generic   docker://1.13.1
ctl03.****   Ready    master   4d3h   v1.12.3-2+bc2e990ce4e046   192.168.10.94    10.13.250.32   Ubuntu 16.04.1 LTS   4.4.0-36-generic   docker://1.13.1```

[zetaab]

imo this is not openstack specific issue, it is same issue in all cloudproviders. So instead of solving this issue in openstack cloud.conf it should be solved in kubernetes node level.

[mape90]

With only two networks you could define PublicNetworkName with the network that you do not want to be internal.

However this fails again when you have 3 or more nics. So real solution for multiple nics would be to have configurable variable "InternalNetworkName" (just example) what would be used to define Internal network. Then all other networks would be Externals.

It seems that at least gce, cloudstack and azure just hardcode that first interface is always the internal one. In aws they seems to be ended in conclusion that the there should also be only one InternalIP (kubernetes/kubernetes#61921).

@zetaab, I do not think this problem would be solvable on generic cloud provider code as there isn't enough information about which of the internal networks would be the "corrrect". And if you add some new information to the networks it would mean that all cloud providers would need to adapt to that. Easiest solution seems to be that there is only one InternalIP and this is how most of the providers are already now working. In openstack case as we do not get the indexes of the networks so we can not just select the first, so we need to have some user defined variable (InternalNetworkName) for selecting the correct network. Also if you have explicit way for selecting the network the you may have more complex network configurations.

[jichenjc]

The way to provide 'ip to used inside k8s cluster' seems to be a valid request... e.g
we have management network, data network , public network allowed in neutron
so I think default to 'internal k8s network first' and if not given, use network[0] is a backward compatible choice

[mape90]

The network[0] would be nice if it would be possible. However we just get list of IPs what is then converted to map and iterated. So there is no guarantee that IPs will be in same order for all nodes. I am not even sure if the list that we get from gophercloud is sorted. But most likely most current users are just using single network or dual network with PublicNetworkName. So as long as those work as previously nothing should break.

[elgamal2020]

What is the setting of external_openstack_network_public_networks: ?

The floating ip Network ?

[lingxiankong]

What is the setting of external_openstack_network_public_networks: ?

The floating ip Network ?

I am not sure where is external_openstack_network_public_networks coming from.

[kayrus]

I think this issue was covered by #884 and #1041