Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux bridge ports are not reflecting the desired state for vlan IDs #631

Closed
etcshad0vv opened this issue Nov 11, 2020 · 6 comments
Closed

Comments

@etcshad0vv
Copy link

etcshad0vv commented Nov 11, 2020

What happened:

Create a Linux vlan-aware(vlan_filtering 1) bridge, and then create either a trunk or access port.

What you expected to happen:

Bridge ports to reflect the desired vlan ids or range in bridge -c vlan show dev output

How to reproduce it (as minimally and precisely as possible):
Apply below desiredState using NNCP:

      interfaces:
      - bridge:
          options:
            stp:
              enabled: false
          port:
          - name: dummy0
            vlan:
              enable-native: true
              mode: access
              tag: 867
        description: Linux bridge for VM connectivity for VLAN 867
        ipv4:
          dhcp: false
          enabled: false
        name: bridge0
        state: up
        type: linux-bridge
      - description: VLAN sub-interface using dummy0
        name: dummy0
        state: up
        type: dummy
        ipv4:
          enabled: false

Anything else we need to know?:

  • Using nmstatectl container from daemonset running on OpenShift cluster and providing the above desired state through yaml works fine!

  • Using nmcli directly from node works fine as well.

  • Using NNCP resource does not reflect right vlan ids(it always shows 2-4094 unless you invoke nmcli con up on the bridge port.(check below)

bridge -c vlan show dev dummy0
dummy0   1 PVID Egress Untagged
         2-4094

Environment:

  • NodeNetworkState on affected nodes (use kubectl get nodenetworkstate <node_name> -o yaml):
oc get nns wdcbmocp3.ocpbm.ibmsdn.local -o yaml
apiVersion: nmstate.io/v1alpha1
kind: NodeNetworkState
metadata:
  creationTimestamp: "2020-09-15T14:29:34Z"
  generation: 1
  managedFields:
  - apiVersion: nmstate.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
          .: {}
          k:{"uid":"25310543-9921-49af-ab49-036e0bc75f3f"}:
            .: {}
            f:apiVersion: {}
            f:kind: {}
            f:name: {}
            f:uid: {}
      f:status:
        .: {}
        f:currentState:
          .: {}
          f:dns-resolver:
            .: {}
            f:config:
              .: {}
              f:search: {}
              f:server: {}
            f:running:
              .: {}
              f:search: {}
              f:server: {}
          f:interfaces: {}
          f:route-rules:
            .: {}
            f:config: {}
          f:routes:
            .: {}
            f:config: {}
            f:running: {}
        f:lastSuccessfulUpdateTime: {}
    manager: kubernetes-nmstate
    operation: Update
    time: "2020-11-11T15:08:29Z"
  name: wdcbmocp3.ocpbm.ibmsdn.local
  ownerReferences:
  - apiVersion: v1
    kind: Node
    name: wdcbmocp3.ocpbm.ibmsdn.local
    uid: 25310543-9921-49af-ab49-036e0bc75f3f
  resourceVersion: "64208805"
  selfLink: /apis/nmstate.io/v1alpha1/nodenetworkstates/wdcbmocp3.ocpbm.ibmsdn.local
  uid: f27f1d43-0b51-44d6-b524-34b50068cdb6
status:
  currentState:
    dns-resolver:
      config:
        search: []
        server:
        - 10.170.30.170
        - 10.170.30.170
      running:
        search: []
        server:
        - 10.170.30.170
        - 10.170.30.170
    interfaces:
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 12:89:2E:00:1E:AF
      mtu: 1400
      name: 0830fe9f4b3ba3d
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 46:21:7C:A9:08:97
      mtu: 1400
      name: 0d95520f48412c9
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 92:D2:AC:11:2E:8B
      mtu: 1400
      name: 0e062e83d0250c4
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 5E:6D:D1:F3:69:A3
      mtu: 1400
      name: 113dd83cedda717
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: EA:A7:3D:CA:2C:1A
      mtu: 1400
      name: 1b8e4a8aaca0e70
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: AA:1B:12:EE:A8:AC
      mtu: 1400
      name: 25ae00938f53147
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 42:12:13:55:18:80
      mtu: 1400
      name: 28d98de35b24f9a
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 32:71:53:19:85:8A
      mtu: 1400
      name: 291a183e5e525ad
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 5E:89:E0:49:CA:4B
      mtu: 1400
      name: 31bb6ee459c68c3
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: FA:32:8A:4E:03:F4
      mtu: 1400
      name: 38756df3b686756
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 42:C1:3B:03:BF:D4
      mtu: 1400
      name: 3b37fe32e1668fc
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: F6:1F:F4:70:BF:9D
      mtu: 1400
      name: 4138550e8a98364
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 02:9E:E6:B4:BD:8A
      mtu: 1400
      name: 429f3dcae132c1e
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 3E:5B:6D:D4:F0:69
      mtu: 1400
      name: 47836d85bc54d31
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: DE:97:4C:30:A7:9F
      mtu: 1400
      name: 507c4a8159a4561
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 1E:72:D4:DD:1C:17
      mtu: 1400
      name: 572abb714249a83
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 26:90:7A:36:3F:B8
      mtu: 1400
      name: 592eff8d3f15555
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 1A:68:20:12:A8:34
      mtu: 1400
      name: 5cc88313ab6e206
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: A6:3B:F6:C7:6C:F1
      mtu: 1400
      name: 7ce4249692fc871
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: BA:E4:AE:1A:00:44
      mtu: 1400
      name: 7d47d79d615b5ba
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 8A:5D:25:EB:1E:BF
      mtu: 1400
      name: 85b4962329ed151
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 9E:8F:8E:97:62:A5
      mtu: 1400
      name: 8782f8313772346
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 6A:3B:4A:9A:CB:FB
      mtu: 1400
      name: 8b0a861dffda7d2
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 96:41:73:4B:B1:28
      mtu: 1400
      name: 926334b96d56c35
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: BA:72:ED:AF:FC:02
      mtu: 1400
      name: 94b6945b5ecf197
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: CE:BE:81:6D:C4:43
      mtu: 1400
      name: 96a0d2b74fc4e1f
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 7E:CD:6E:E1:79:01
      mtu: 1400
      name: acea670292660c9
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 7E:2D:70:D8:1C:12
      mtu: 1400
      name: b20a95efc2b2cb2
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 1A:10:41:C2:F7:14
      mtu: 1400
      name: b9e28c7f7e36b89
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 9A:96:09:89:B0:FF
      mtu: 1400
      name: babc1106df0d4f8
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 42:C0:F1:AE:EF:29
      mtu: 1400
      name: bd79e67cae69d50
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: F2:0E:EC:27:FE:F8
      mtu: 1400
      name: be7d584740623b3
      state: down
      type: ethernet
    - ipv4:
        dhcp: false
        enabled: false
      ipv6:
        enabled: false
      link-aggregation:
        mode: 802.3ad
        options:
          ad_actor_system: "00:00:00:00:00:00"
          miimon: "100"
        slaves:
        - eno1
        - eno3
      mac-address: AC:1F:6B:C8:02:8A
      mtu: 1500
      name: bond0
      state: up
      type: bond
    - ipv4:
        address:
        - ip: 169.45.216.138
          prefix-length: 27
        dhcp: false
        enabled: true
      ipv6:
        address:
        - ip: fe80::ae1f:6bff:fec8:28b
          prefix-length: 64
        autoconf: false
        dhcp: false
        enabled: true
      link-aggregation:
        mode: 802.3ad
        options:
          ad_actor_system: "00:00:00:00:00:00"
          miimon: "100"
        slaves:
        - eno2
        - eno4
      mac-address: AC:1F:6B:C8:02:8B
      mtu: 1500
      name: bond1
      state: up
      type: bond
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: de:45:7c:8d:85:45
      mtu: 1400
      name: br-int
      state: down
      type: ovs-interface
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 0e:fd:15:19:30:4d
      mtu: 1400
      name: br-local
      state: down
      type: ovs-interface
    - bridge:
        options:
          group-forward-mask: 0
          mac-ageing-time: 300
          multicast-snooping: true
          stp:
            enabled: false
            forward-delay: 15
            hello-time: 2
            max-age: 20
            priority: 32768
        port:
        - name: bond0
          stp-hairpin-mode: false
          stp-path-cost: 100
          stp-priority: 32
          vlan:
            enable-native: false
            mode: trunk
            trunk-tags:
            - id: 846
            - id: 888
        - name: vethf7e02a7f
          stp-hairpin-mode: false
          stp-path-cost: 2
          stp-priority: 32
        - name: veth0488e629
          stp-hairpin-mode: false
          stp-path-cost: 2
          stp-priority: 32
      ipv4:
        address:
        - ip: 10.170.30.164
          prefix-length: 26
        dhcp: false
        enabled: true
      ipv6:
        address:
        - ip: fe80::db43:3d46:fa42:bf17
          prefix-length: 64
        autoconf: false
        dhcp: false
        enabled: true
      mac-address: 76:F8:64:F1:42:FA
      mtu: 1500
      name: br-netvm
      state: up
      type: linux-bridge
    - bridge:
        options:
          group-forward-mask: 0
          mac-ageing-time: 300
          multicast-snooping: true
          stp:
            enabled: false
            forward-delay: 15
            hello-time: 2
            max-age: 20
            priority: 32768
        port:
        - name: dummy0
          stp-hairpin-mode: false
          stp-path-cost: 100
          stp-priority: 32
          vlan:
            mode: access
            tag: 867
            trunk-tags: []
      description: Linux bridge for VM connectivity for VLAN 867
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        autoconf: false
        dhcp: false
        enabled: false
      mac-address: FE:77:F8:CE:F3:32
      mtu: 1500
      name: br867
      state: up
      type: linux-bridge
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 42:43:84:62:2E:52
      mtu: 1400
      name: c114380e8193303
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 4A:4C:5C:C3:ED:C0
      mtu: 1400
      name: c4acaa36b4c0d9a
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 36:D9:C3:C7:C1:C6
      mtu: 1400
      name: c5c193a8fa0ddbb
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 2E:88:AA:C2:64:27
      mtu: 1400
      name: d1268c64d9d4708
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 2E:A1:95:84:71:65
      mtu: 1400
      name: d80b54b988a5614
      state: down
      type: ethernet
    - description: VLAN sub-interface using dummy0
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        autoconf: false
        dhcp: false
        enabled: false
      mac-address: FE:77:F8:CE:F3:32
      mtu: 1500
      name: dummy0
      state: up
      type: dummy
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: F2:3C:54:79:55:8E
      mtu: 1400
      name: e1526f684facc19
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 16:77:12:C4:7E:EB
      mtu: 1400
      name: ecf134e8ef41d45
      state: down
      type: ethernet
    - ethernet:
        auto-negotiation: true
        duplex: full
        speed: 10000
        sr-iov:
          total-vfs: 0
          vfs: []
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        autoconf: false
        dhcp: false
        enabled: false
      mac-address: AC:1F:6B:C8:02:8A
      mtu: 1500
      name: eno1
      state: up
      type: ethernet
    - ethernet:
        auto-negotiation: true
        duplex: full
        speed: 10000
        sr-iov:
          total-vfs: 0
          vfs: []
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        autoconf: false
        dhcp: false
        enabled: false
      mac-address: AC:1F:6B:C8:02:8B
      mtu: 1500
      name: eno2
      state: up
      type: ethernet
    - ethernet:
        auto-negotiation: true
        duplex: full
        speed: 10000
        sr-iov:
          total-vfs: 0
          vfs: []
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        autoconf: false
        dhcp: false
        enabled: false
      mac-address: AC:1F:6B:C8:02:8A
      mtu: 1500
      name: eno3
      state: up
      type: ethernet
    - ethernet:
        auto-negotiation: true
        duplex: full
        speed: 10000
        sr-iov:
          total-vfs: 0
          vfs: []
      ipv4:
        dhcp: false
        enabled: false
      ipv6:
        autoconf: false
        dhcp: false
        enabled: false
      mac-address: AC:1F:6B:C8:02:8B
      mtu: 1500
      name: eno4
      state: up
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 36:BD:D4:BE:70:69
      mtu: 1400
      name: fa54b2fc0f6cdf1
      state: down
      type: ethernet
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 2A:F7:6A:88:26:F9
      mtu: 65000
      name: genev_sys_6081
      state: down
      type: unknown
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mtu: 65536
      name: lo
      state: down
      type: unknown
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: 00:00:a9:fe:21:01
      mtu: 1400
      name: ovn-k8s-gw0
      state: down
      type: ovs-interface
    - ipv4:
        enabled: false
      ipv6:
        enabled: false
      mac-address: d2:49:51:15:10:ae
      mtu: 1400
      name: ovn-k8s-mp0
      state: down
      type: ovs-interface
    route-rules:
      config: []
    routes:
      config:
      - destination: 0.0.0.0/0
        metric: -1
        next-hop-address: 169.45.216.129
        next-hop-interface: bond1
        table-id: 0
      running:
      - destination: 0.0.0.0/0
        metric: 301
        next-hop-address: 169.45.216.129
        next-hop-interface: bond1
        table-id: 254
      - destination: 169.45.216.128/27
        metric: 301
        next-hop-address: ""
        next-hop-interface: bond1
        table-id: 254
      - destination: 10.170.30.128/26
        metric: 425
        next-hop-address: ""
        next-hop-interface: br-netvm
        table-id: 254
      - destination: fe80::/64
        metric: 301
        next-hop-address: ""
        next-hop-interface: bond1
        table-id: 254
      - destination: fe80::/64
        metric: 425
        next-hop-address: ""
        next-hop-interface: br-netvm
        table-id: 254
      - destination: ff00::/8
        metric: 256
        next-hop-address: ""
        next-hop-interface: dummy0
        table-id: 255
  lastSuccessfulUpdateTime: "2020-11-11T15:08:29Z"
  • Problematic NodeNetworkConfigurationPolicy:
apiVersion: v1
items:
- apiVersion: nmstate.io/v1alpha1
  kind: NodeNetworkConfigurationPolicy
  metadata:
    name: bridge0-create
  spec:
    desiredState:
      interfaces:
      - bridge:
          options:
            stp:
              enabled: false
          port:
          - name: dummy0
            vlan:
              enable-native: true
              mode: access
              tag: 867
        description: Linux bridge for VM connectivity for VLAN 867
        ipv4:
          dhcp: false
          enabled: false
        name: bridge0
        state: up
        type: linux-bridge
      - description: VLAN sub-interface using dummy0
        name: dummy0
        state: up
        type: dummy
        ipv4:
          enabled: false
      routes:
        config:
        - destination: 0.0.0.0/0
          next-hop-address: 169.45.216.129
          next-hop-interface: bond1
    nodeSelector:
      node-role.kubernetes.io/worker: ""
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""
  • kubernetes-nmstate image (use kubectl get pods --all-namespaces -l app=kubernetes-nmstate -o jsonpath='{.items[0].spec.containers[0].image}'):
oc get pods --all-namespaces -l app=kubernetes-nmstate -o jsonpath='{.items[0].spec.containers[0].image}'
registry.redhat.io/container-native-virtualization/kubernetes-nmstate-handler-rhel8@sha256:d25fe2735181b8bec769d87f37fe8b3eef23fc11f009e4b63c6fe96b83f0d838
  • NetworkManager version (use nmcli --version)
nmcli --version
nmcli tool, version 1.22.8-5.el8_2
  • Kubernetes version (use kubectl version):
oc version
Client Version: 4.5.0-202005291417-9933eb9
Server Version: 4.5.13
Kubernetes Version: v1.18.3+47c0e71
  • OS (e.g. from /etc/os-release):
cat /etc/os-release
NAME="Red Hat Enterprise Linux CoreOS"
VERSION="45.82.202009181447-0"
VERSION_ID="4.5"
OPENSHIFT_VERSION="4.5"
RHEL_VERSION="8.2"
PRETTY_NAME="Red Hat Enterprise Linux CoreOS 45.82.202009181447-0 (Ootpa)"
ID="rhcos"
ID_LIKE="rhel fedora"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:8::coreos"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="OpenShift Container Platform"
REDHAT_BUGZILLA_PRODUCT_VERSION="4.5"
REDHAT_SUPPORT_PRODUCT="OpenShift Container Platform"
REDHAT_SUPPORT_PRODUCT_VERSION="4.5"
OSTREE_VERSION='45.82.202009181447-0'
  • Others:
@qinqon
Copy link
Member

qinqon commented Nov 11, 2020

Hi, vlan filtering is not supported at kubernetes-nmstate yet, we just do a best effort [1] to bridge-cni vlan feature [2] to work fine.

[1] https://github.com/nmstate/kubernetes-nmstate/blob/master/build/bin/vlan-filtering
[2] https://github.com/containernetworking/plugins/blob/master/plugins/main/bridge/README.md

@etcshad0vv
Copy link
Author

@qinqon thanks for your prompt response.
To understand this, you mean that it doesn't work properly when nmstate is used to enslave ports however it works when ports are created by net-attach-def using multus multi-net plugin?

@qinqon
Copy link
Member

qinqon commented Nov 11, 2020

The nmstate is fine issue is at kubernetes-nmstate since we try to do vlan filtering on the whole range so whatever is configured with net-attach-def + bridge CNI will work fine.

@etcshad0vv
Copy link
Author

aha i see, i saw that vlan-filtering script invoked on the golang source, so if nmstate can handle state properly with respective vlan ids defined in the desiredState, i wonder why you still use the vlan-filtering binary which pretty much is messing the right state applied by nmstatectl.
If i would use net-attach-def and bridge-cni, is there some way to define a local interface(e.g eth0) to act as trunk port ?

@phoracek
Copy link
Member

Our intention was to configure all Linux bridges in a way that they treat their port as a trunk. Users then can simply create a net-attach-def with a VLAN set, connect to the bridge and enjoy the traffic passed up and down. Back at the time vlan-filtering was not yet available in nmstate.

Now when vlan-filtering is available in nmstate, we'd like to migrate to it and allow config such as the one you shared in this issue. We haven't done so due to capacity issues. It is not as simple as dropping the script, we need to maintain compatibility for people upgrading from the older version. I guess we should apply the previous default unless our users specify their own vlan-tagging config.

There is a tracker for it, in case somebody would want to contribute the feature. I'd be glad to guide the effort: #171

If i would use net-attach-def and bridge-cni, is there some way to define a local interface(e.g eth0) to act as trunk port ?

That is what should happen automatically - all ports assigned to a bridge via the desired state should become trunks.

@phoracek
Copy link
Member

Closing in favor of #634

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants