Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ovn lb select the local chassis's backend prefer #4894

Open
wants to merge 8 commits into
base: master
Choose a base branch
from

Conversation

changluyi
Copy link
Collaborator

@changluyi changluyi commented Jan 2, 2025

Pull Request

What type of this PR

Examples of user facing changes:

  • Features
  • Bug fixes
  • Docs
  • Tests

ovn 实现原理。

OVN 这部分代码修改原理:

实现原理是获取到vip backend的 chassis , 新增加流表入下:

is_chassis_resident("nginx-6f55f97f94-8kvlx.default") , 其中 nginx-6f55f97f94-8kvlx.default 是某个vip 的backend下的任意一个lsp,backends=10.16.0.3:80 表示这个lb 在这个chassis下的所有backendip: port 。

  table=13(ls_in_lb           ), priority=140  , match=(ct.new && ip4.dst == 1.1.1.1 && tcp.dst == 80 && is_chassis_resident("nginx-6f55f97f94-8kvlx.default")), action=(ct_lb_mark(backends=10.16.0.3:80; hash_fields="ip_src");)
  table=13(ls_in_lb           ), priority=140  , match=(ct.new && ip4.dst == 1.1.1.1 && tcp.dst == 80 && is_chassis_resident("nginx-6f55f97f94-zlr4p.default")), action=(ct_lb_mark(backends=10.16.0.2:80; hash_fields="ip_src");)

如果hit不到以上流表,就会走原来的ovn lb 规则。

该功能需要配置Load_balancer的 两个字段
option:prefer_local_backend

option:prefer_local_backend=true

ip_port_mappings

"10.16.0.10"="nginx-6f55f97f94-m4bxx.default:10.16.0.4", "10.16.0.2"="nginx-6f55f97f94-zlr4p.default:10.16.0.4", "10.16.0.3"="nginx-6f55f97f94-8kvlx.default:10.16.0.4", "10.16.0.5"="nginx-6f55f97f94-ns9th.default:10.16.0.4"

kube-ovn 增加场景

场景1:switchlbRules 支持 OVN LB 优先走本地 backend

使用方法:

kube-ovn-controller 开启开关 --enable-ovn-lb-prefer-local = true

用例:

kind: SwitchLBRule
metadata:
  name:  cjh-slr-nginx
spec:
  vip: 1.1.1.1
  sessionAffinity: ClientIP
  namespace: default
  selector:
    - app:nginx
  ports:
  - name: http
    port: 80
    targetPort: 80
    protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: "app"
                    operator: In
                    values:
                      - nginx
              topologyKey: "kubernetes.io/hostname"
      containers:
      - name: nginx-container
        image: nginx:alpine
        imagePullPolicy: IfNotPresent
        ports:
        - containerPort: 80

场景2: ovn lb 模式下支持 internalTrafficPolicy: Local , 真正实现发送本地 backend

使用方法:

kube-ovn-controller 开启开关 --enable-ovn-lb-prefer-local = true

用例:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app-container
        image: nginx:alpine
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: my-app-service
spec:
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 80
    protocol: TCP
  internalTrafficPolicy: Local
  type: ClusterIP

场景3:underlay 模式下 和 metallb 一起使用, 该场景主要是为了metallb 模式下支持某个underlay subnet提供外部地址池的vip, vip的后端pod同样在该子网下, 并保证clientIP 不被改变。

流量走向:

image

使用方法:

  1. kube-ovn-controller 开启开关 --enable-ovn-lb-prefer-local = true
  2. subnet.spec.enableExternalLBAddress = true
  3. subnet.spec.excludeIps 添加外部地址池的地址段。
  4. service 配置 externalTrafficPolicy:Local

Which issue(s) this PR fixes

Fixes #(issue-number)

@dosubot dosubot bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Jan 2, 2025
@coveralls
Copy link

coveralls commented Jan 2, 2025

Pull Request Test Coverage Report for Build 13255351680

Warning: This coverage report may be inaccurate.

This pull request's base commit is no longer the HEAD commit of its target branch. This means it includes changes from outside the original pull request, including, potentially, unrelated coverage changes.

Details

  • 0 of 446 (0.0%) changed or added relevant lines in 8 files are covered.
  • 3 unchanged lines in 2 files lost coverage.
  • Overall coverage decreased (-0.2%) to 22.115%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pkg/controller/config.go 0 2 0.0%
pkg/controller/init.go 0 5 0.0%
pkg/ovs/ovn-nb-load_balancer.go 0 32 0.0%
pkg/controller/endpoint.go 0 36 0.0%
pkg/controller/service.go 0 40 0.0%
pkg/daemon/controller.go 0 45 0.0%
pkg/ovs/ovs-ofctl.go 0 117 0.0%
pkg/daemon/controller_linux.go 0 169 0.0%
Files with Coverage Reduction New Missed Lines %
pkg/controller/init.go 1 0.0%
pkg/ovs/ovn-nb-bfd.go 2 61.61%
Totals Coverage Status
Change from base Build 13236788436: -0.2%
Covered Lines: 10376
Relevant Lines: 46919

💛 - Coveralls

@changluyi changluyi force-pushed the lb_service_select_local_backend branch from 93b8ed4 to 318cff3 Compare January 6, 2025 03:06
@changluyi changluyi changed the title template test ovn lb select the local chassis's backend prefer Jan 6, 2025
@changluyi changluyi added feature New network feature enhancement Improve exist functions labels Jan 6, 2025
@changluyi changluyi force-pushed the lb_service_select_local_backend branch from 318cff3 to cdbf329 Compare January 7, 2025 11:30
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Jan 7, 2025
@changluyi changluyi force-pushed the lb_service_select_local_backend branch from cdbf329 to ffd7994 Compare January 8, 2025 02:47
@dosubot dosubot bot added size:XS This PR changes 0-9 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Jan 8, 2025
@changluyi changluyi force-pushed the lb_service_select_local_backend branch from ffd7994 to 21e47a8 Compare January 9, 2025 05:30
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Jan 9, 2025
@changluyi changluyi force-pushed the lb_service_select_local_backend branch from 21e47a8 to 2ade8ff Compare January 9, 2025 05:31
@dosubot dosubot bot added size:XS This PR changes 0-9 lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Jan 9, 2025
@changluyi changluyi force-pushed the lb_service_select_local_backend branch from 2ade8ff to add2969 Compare January 9, 2025 06:13
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Jan 15, 2025
@changluyi changluyi force-pushed the lb_service_select_local_backend branch 2 times, most recently from 8a81fc8 to 4bd7410 Compare January 20, 2025 09:43
@dosubot dosubot bot added size:XXL This PR changes 1000+ lines, ignoring generated files. and removed size:XL This PR changes 500-999 lines, ignoring generated files. labels Jan 23, 2025
@changluyi changluyi force-pushed the lb_service_select_local_backend branch from a703a56 to d1385c6 Compare January 24, 2025 02:10
@changluyi changluyi requested a review from zhangzujian January 24, 2025 02:24
@changluyi changluyi force-pushed the lb_service_select_local_backend branch 2 times, most recently from 48a02a6 to 692ec96 Compare February 5, 2025 03:06
@changluyi changluyi force-pushed the lb_service_select_local_backend branch from 692ec96 to c24b4c3 Compare February 5, 2025 03:30
@zhangzujian
Copy link
Member

The following occasional failure need to be fixed:

  [FAILED] Code Location: /home/zhang/kube-ovn/test/e2e/metallb/e2e_test.go:417
  Expected
      <string>: kube-ovn-control-plane
  to equal
      <string>: kube-ovn-worker
  In [It] at: /home/zhang/kube-ovn/test/e2e/metallb/e2e_test.go:358 @ 02/06/25 02:45:06.75

@changluyi
Copy link
Collaborator Author

The following occasional failure need to be fixed:

  [FAILED] Code Location: /home/zhang/kube-ovn/test/e2e/metallb/e2e_test.go:417
  Expected
      <string>: kube-ovn-control-plane
  to equal
      <string>: kube-ovn-worker
  In [It] at: /home/zhang/kube-ovn/test/e2e/metallb/e2e_test.go:358 @ 02/06/25 02:45:06.75

I tested in my local env and github CI flow, but not find this error

@zbb88888
Copy link
Collaborator

zbb88888 commented Feb 7, 2025

matallb 也可以使用 kube-ovn unerlay subnet 中 exclude 中的 ip:metallb 使用 br-provider 网桥么, 这样 metallb 可以将流量转到 kube-ovn underlay subnet pod 内, 这样子使用吗?

@changluyi
Copy link
Collaborator Author

matallb 也可以使用 kube-ovn unerlay subnet 中 exclude 中的 ip:metallb 使用 br-provider 网桥么, 这样 metallb 可以将流量转到 kube-ovn underlay subnet pod 内, 这样子使用吗?

是的

Signed-off-by: clyi <[email protected]>
Signed-off-by: clyi <[email protected]>
Signed-off-by: clyi <[email protected]>
@changluyi changluyi force-pushed the lb_service_select_local_backend branch from 5984750 to d091293 Compare February 10, 2025 14:06
Signed-off-by: clyi <[email protected]>
@changluyi changluyi force-pushed the lb_service_select_local_backend branch from 0ce2f62 to 4250649 Compare February 11, 2025 02:38
@changluyi changluyi force-pushed the lb_service_select_local_backend branch from 2e7a692 to eb38483 Compare February 11, 2025 07:47
@changluyi
Copy link
Collaborator Author

The following occasional failure need to be fixed:

  [FAILED] Code Location: /home/zhang/kube-ovn/test/e2e/metallb/e2e_test.go:417
  Expected
      <string>: kube-ovn-control-plane
  to equal
      <string>: kube-ovn-worker
  In [It] at: /home/zhang/kube-ovn/test/e2e/metallb/e2e_test.go:358 @ 02/06/25 02:45:06.75

It should be a problem with the l2servicestatus field of metallb. It seems that the status.node of this field is not necessarily the node where the vip is located. It is probably a metallb bug.

@zhangzujian
Copy link
Member

The following occasional failure need to be fixed:

  [FAILED] Code Location: /home/zhang/kube-ovn/test/e2e/metallb/e2e_test.go:417
  Expected
      <string>: kube-ovn-control-plane
  to equal
      <string>: kube-ovn-worker
  In [It] at: /home/zhang/kube-ovn/test/e2e/metallb/e2e_test.go:358 @ 02/06/25 02:45:06.75

It should be a problem with the l2servicestatus field of metallb. It seems that the status.node of this field is not necessarily the node where the vip is located. It is probably a metallb bug.

If the l2servicestatus field may be incorrect, we should not use it in our e2e tests.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Improve exist functions feature New network feature size:XXL This PR changes 1000+ lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants