Duplicate IPs - how to debug?

[sgpinkus]

Observing pods with the same pod IP address in k8s cluster using flannel. I read here that it's apparently not flannel's job to assign IP addresses. But I'm at a loss how to fix, not sure how IP addresses actually get assigned but it's in flannels subnet so not completely unrelated to flannel. There is no clues in the troubleshooting guide.

Expected Behavior

Each pod has unique IP address

Current Behavior

Pods with same IP address

Possible Solution

No idea! Give me a hint please.

Steps to Reproduce (for bugs)

1. Setting up cluster on VM as described here.
2. Install flannel:

kubectl patch node test-vm{4..6} -p '{ "spec": { "podCIDR": "10.244.0.0/16" } }' # Had to do this to get flannel to start.
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

3. Start some pods, observe IPs:

NAMESPACE     NAME                               READY   STATUS    RESTARTS   AGE     IP           NODE       NOMINATED NODE   READINESS GATES
default       curl                               1/1     Running   0          14m     10.244.0.4   test-vm6   <none>           <none>
default       my-nginx-cf54cdbf7-d6s9m           1/1     Running   0          17m     10.244.0.3   test-vm6   <none>           <none>
default       my-nginx-cf54cdbf7-twrvw           1/1     Running   0          17m     10.244.0.2   test-vm6   <none>           <none>
default       my-nginx-cf54cdbf7-xpff6           1/1     Running   0          17m     10.244.0.4   test-vm5   <none>           <none>
default       my-nginx-more-5f79688b9d-4c9jk     1/1     Running   0          3m10s   10.244.0.6   test-vm5   <none>           <none>
default       my-nginx-more-5f79688b9d-7htsn     1/1     Running   0          3m18s   10.244.0.5   test-vm5   <none>           <none>
default       my-nginx-more-5f79688b9d-gqz9b     1/1     Running   0          3m4s    10.244.0.7   test-vm5   <none>           <none>
default       nginx1                             1/1     Running   0          9s      10.244.0.8   test-vm5   <none>           <none>
kube-system   coredns-64897985d-kt82d            1/1     Running   0          41m     10.244.0.2   test-vm5   <none>           <none>
kube-system   coredns-64897985d-rd7gz            1/1     Running   0          41m     10.244.0.3   test-vm5   <none>           <none>
kube-system   etcd-test-vm4                      1/1     Running   0          41m     10.1.4.36    test-vm4   <none>           <none>
kube-system   kube-apiserver-test-vm4            1/1     Running   0          41m     10.1.4.36    test-vm4   <none>           <none>
kube-system   kube-controller-manager-test-vm4   1/1     Running   0          41m     10.1.4.36    test-vm4   <none>           <none>
kube-system   kube-flannel-ds-snkhk              1/1     Running   0          29m     10.1.4.38    test-vm6   <none>           <none>
kube-system   kube-flannel-ds-wtmqg              1/1     Running   0          29m     10.1.4.37    test-vm5   <none>           <none>
kube-system   kube-flannel-ds-x46xw              1/1     Running   0          29m     10.1.4.36    test-vm4   <none>           <none>
kube-system   kube-proxy-mjl69                   1/1     Running   0          41m     10.1.4.37    test-vm5   <none>           <none>
kube-system   kube-proxy-vz2p2                   1/1     Running   0          41m     10.1.4.36    test-vm4   <none>           <none>
kube-system   kube-proxy-xg4gg                   1/1     Running   0          41m     10.1.4.38    test-vm6   <none>           <none>
kube-system   kube-scheduler-test-vm4            1/1     Running   0          41m     10.1.4.36    test-vm4   <none>           <none>

Context

Pod networking is not working.

Your Environment

• Flannel version: https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml
• Backend used (e.g. vxlan or udp): vxlan
• Etcd version: -
• Kubernetes version (if used): v.1.21.11
• Operating System and version: Linux test-vm4 4.19.0-19-amd64 write a prototype :) #1 SMP Debian 4.19.232-1 (2022-03-07) x86_64 GNU/Linux
• Link to your project (optional):

[sgpinkus]

Closing - made PR to update troubleshooting guide once fixed issue.