How to correctly create Overlay and Underlay networks in the same cluster?

[renyunkang]

When the Overlay and Underlay networks are created in the same cluster, how to configure the clustercidr of kube-proxy?

I am encountering an issue when the clustercidr is set to 10.233.64.0/18, the underlay network is configured properly, a subnet 172.31.15.0/24 with vlan id 15 is created, but when a pod uses this to access the Kubernetes API, an error occurs:

root@node22:~# kubectl logs -f -n kube-system openebs-localpv-provisioner-68c878c769-zgccq
I1019 02:42:10.320787       1 start.go:66] Starting Provisioner...
failure in preupgrade tasks: failed to list localpv based pv(s): Get "https://10.233.0.1:443/api/v1/persistentvolumes?labelSelector=openebs.io%2Fcas-type%3Dlocal-device": read tcp 172.31.15.11:40016->10.233.0.1:443: read: connection reset by peer

The key points are:

• Cluster CIDR is set to 10.233.64.0/18 for the overlay network
• A VLAN subnet 172.31.15.0/24 is created for the underlay
• Pods using the VLAN subnet can't access the Kubernetes API and get connection reset errors 

Could you please help me find out if there is something wrong with the configuration?

[hhyasdf]

Hi @renyunkang, I'm wondering if the service CIDR is overlapped with your overlay pod CIDR? That might cause problem like this.

[renyunkang]

They do not overlap.

The overlay pod CIDR (10.233.64.0/18) and the service CIDR (10.233.0.0/18) do not overlap in this case.

[hhyasdf]

@renyunkang，If I'm right, the endpoint behind cluster ip is the kube-apiserver-node22 (172.31.50.22). So it's not a problem between overlay and underlay, but is a problem that vlan pods cannot access a host ip through a cluster ip.

Could you please check if the vlan pod can access the endpoint directly (not through the cluster ip) and show us the kube-proxy configuration?

[renyunkang]

You are right, the vlan pod can access the endpoint directly.

root@node22:~/hybridnet# kubectl get pod -o wide
NAME                            READY   STATUS    RESTARTS   AGE     IP            NODE     NOMINATED NODE   READINESS GATES
tools-network-9fc4448c8-888nn   1/1     Running   0          2m39s   172.31.15.3   node23   <none>           <none>
root@node22:~/hybridnet# 
root@node22:~/hybridnet# 
root@node22:~/hybridnet# kubectl exec -it tools-network-9fc4448c8-888nn -- ping 172.31.50.22
PING 172.31.50.22 (172.31.50.22): 56 data bytes
64 bytes from 172.31.50.22: seq=0 ttl=62 time=26.353 ms
64 bytes from 172.31.50.22: seq=1 ttl=62 time=0.349 ms
64 bytes from 172.31.50.22: seq=2 ttl=62 time=0.392 ms
^C
--- 172.31.50.22 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.349/9.031/26.353 ms
root@node22:~/hybridnet# kubectl exec -it tools-network-9fc4448c8-888nn -- curl -k https://10.233.0.1:443
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 10.233.0.1:443 
command terminated with exit code 35
root@node22:~/hybridnet# kubectl exec -it tools-network-9fc4448c8-888nn -- curl -k https://172.31.50.22:6443
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {},
  "code": 403
}

Here is the configuration of kube-proxy:

root@node22:~/hybridnet# kubectl get cm -n kube-system kube-proxy -o yaml
apiVersion: v1
data:
  config.conf: |-
    apiVersion: kubeproxy.config.k8s.io/v1alpha1
    bindAddress: 0.0.0.0
    bindAddressHardFail: false
    clientConnection:
      acceptContentTypes: ""
      burst: 0
      contentType: ""
      kubeconfig: /var/lib/kube-proxy/kubeconfig.conf
      qps: 0
    clusterCIDR: 10.233.64.0/18
    configSyncPeriod: 0s
    conntrack:
      maxPerCore: null
      min: null
      tcpCloseWaitTimeout: null
      tcpEstablishedTimeout: null
    detectLocalMode: ""
    enableProfiling: false
    healthzBindAddress: ""
    hostnameOverride: ""
    iptables:
      masqueradeAll: false
      masqueradeBit: 14
      minSyncPeriod: 0s
      syncPeriod: 30s
    ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: ""
      strictARP: false
      syncPeriod: 0s
      tcpFinTimeout: 0s
      tcpTimeout: 0s
      udpTimeout: 0s
    kind: KubeProxyConfiguration
    metricsBindAddress: ""
    mode: ipvs
    nodePortAddresses: null
    oomScoreAdj: null
    portRange: ""
    showHiddenMetricsForVersion: ""
    udpIdleTimeout: 0s
    winkernel:
      enableDSR: false
      networkName: ""
      sourceVip: ""
  kubeconfig.conf: |-
    apiVersion: v1
    kind: Config
    clusters:
    - cluster:
        certificate-authority: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        server: https://lb.kubesphere.local:6443
      name: default
    contexts:
    - context:
        cluster: default
        namespace: default
        user: default
      name: default
    current-context: default
    users:
    - name: default
      user:
        tokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
kind: ConfigMap
metadata:
  creationTimestamp: "2023-10-20T01:52:16Z"
  labels:
    app: kube-proxy
  name: kube-proxy
  namespace: kube-system
  resourceVersion: "298"
  uid: 2905beb9-8b4c-4515-8b2a-fad5e89da4c5

[hhyasdf]

There might be some problems with kube-proxy configuration. What's the masqueradeBit: 14 for? Can you try to remove it?

[renyunkang]

It should be automatically generated by kubeadm.
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-proxy/

[renyunkang]

But after I removed masqueradeBit, it worked fine. So what value should I set to?

[hhyasdf]

Don't use it unless you really need :).