Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No SRIOV devices available #609

Open
gavmckee80 opened this issue Nov 28, 2024 · 6 comments
Open

No SRIOV devices available #609

gavmckee80 opened this issue Nov 28, 2024 · 6 comments

Comments

@gavmckee80
Copy link

What happened?

Resources failed to be discovered on a node

apiVersion: v1
kind: ConfigMap
metadata:
  name: sriovdp-config
  namespace: kube-system
data:
  config.json: |
    {
        "resourceList": [
            {
                "resourceName": "mlnx_sriov_cx7",
                "resourcePrefix": "mellanox",
                "selectors": {
                    "vendors": ["15b3"],
                    "devices": ["101e"],
                    "drivers": ["mlx5_core"],
                    "pciAddresses": ["0000:42:00.0", "0000:42:00.1"]
                }
            }
        ]
    }

Logs from the sriov plugin

kubectl logs kube-sriov-device-plugin-6sr4l -n kube-system
I1128 02:40:11.497892       1 manager.go:57] Using Kubelet Plugin Registry Mode
I1128 02:40:11.498762       1 main.go:46] resource manager reading configs
I1128 02:40:11.498804       1 manager.go:86] raw ResourceList: {
    "resourceList": [
        {
            "resourceName": "mlnx_sriov_cx7",
            "resourcePrefix": "mellanox",
            "selectors": {
                "vendors": ["15b3"],
                "devices": ["101e"],
                "drivers": ["mlx5_core"],
                "pciAddresses": ["0000:42:00.0", "0000:42:00.1"]
            }
        }
    ]
}
I1128 02:40:11.499316       1 factory.go:203] *types.NetDeviceSelectors for resource mlnx_sriov_cx7 is [0xc00088e120]
I1128 02:40:11.499327       1 manager.go:106] unmarshalled ResourceList: [{ResourcePrefix:mellanox ResourceName:mlnx_sriov_cx7 DeviceType:netDevice ExcludeTopology:false Selectors:0xc000012198 AdditionalInfo:map[] SelectorObjs:[0xc00088e120]}]
I1128 02:40:11.499363       1 manager.go:217] validating resource name "mellanox/mlnx_sriov_cx7"
I1128 02:40:11.499368       1 main.go:62] Discovering host devices
WARNING: failed to read int from file: open /sys/devices/system/node/node0/cpu0/online: no such file or directory
I1128 02:40:11.828209       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:23:00.0	02          	Broadcom Inc. and...	NetXtreme BCM5719 Gigabit Ethernet PCIe
I1128 02:40:11.828639       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:23:00.1	02          	Broadcom Inc. and...	NetXtreme BCM5719 Gigabit Ethernet PCIe
I1128 02:40:11.828788       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:23:00.2	02          	Broadcom Inc. and...	NetXtreme BCM5719 Gigabit Ethernet PCIe
I1128 02:40:11.828970       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:23:00.3	02          	Broadcom Inc. and...	NetXtreme BCM5719 Gigabit Ethernet PCIe
I1128 02:40:11.829154       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:00.0	02          	Mellanox Technolo...	MT2910 Family [ConnectX-7]
I1128 02:40:11.895928       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:00.1	02          	Mellanox Technolo...	MT2910 Family [ConnectX-7]
I1128 02:40:11.906113       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:00.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.906319       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:00.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.906543       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:00.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.906810       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:00.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.907062       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:00.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.907268       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:00.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.907474       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:01.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.907734       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:01.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.908059       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:01.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.908387       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:01.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.908774       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:01.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.909093       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:01.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.909499       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:01.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.909815       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:01.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.910199       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:02.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.910547       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:02.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.910825       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:02.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.911088       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:02.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.911282       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:02.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.911468       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:02.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.911668       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:02.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.912068       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:02.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.913131       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:03.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.913467       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:03.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.913665       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:07.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.913869       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:07.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.914064       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:07.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.914305       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:07.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.914584       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:08.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.914841       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:08.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.915060       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:08.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.915282       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:08.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.915535       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:08.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.915749       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:08.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.993911       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:08.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.994274       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:08.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.994516       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:09.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.994701       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:09.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.994920       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:09.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.995105       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:09.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.995342       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:09.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.995560       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:09.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.995738       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:09.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.995956       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:09.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.996170       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:0a.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.996392       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:0a.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.996572       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:0a.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.996786       1 netDeviceProvider.go:67] netdevice AddTargetDevices(): device found: 0000:42:0a.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.996973       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:23:00.0	02          	Broadcom Inc. and...	NetXtreme BCM5719 Gigabit Ethernet PCIe
I1128 02:40:11.996993       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:23:00.1	02          	Broadcom Inc. and...	NetXtreme BCM5719 Gigabit Ethernet PCIe
I1128 02:40:11.996997       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:23:00.2	02          	Broadcom Inc. and...	NetXtreme BCM5719 Gigabit Ethernet PCIe
I1128 02:40:11.997000       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:23:00.3	02          	Broadcom Inc. and...	NetXtreme BCM5719 Gigabit Ethernet PCIe
I1128 02:40:11.997003       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:00.0	02          	Mellanox Technolo...	MT2910 Family [ConnectX-7]
I1128 02:40:11.997007       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:00.1	02          	Mellanox Technolo...	MT2910 Family [ConnectX-7]
I1128 02:40:11.997010       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:00.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997012       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:00.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997015       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:00.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997017       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:00.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997021       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:00.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997023       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:00.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997026       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:01.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997029       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:01.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997031       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:01.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997034       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:01.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997037       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:01.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997039       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:01.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997042       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:01.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997045       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:01.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997047       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:02.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997050       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:02.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997052       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:02.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997055       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:02.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997057       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:02.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997060       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:02.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997063       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:02.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997065       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:02.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997068       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:03.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997070       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:03.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997072       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:07.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997075       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:07.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997078       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:07.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997081       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:07.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997084       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:08.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997087       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:08.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997089       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:08.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997092       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:08.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997095       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:08.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997098       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:08.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997101       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:08.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997103       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:08.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997106       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:09.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997109       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:09.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997111       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:09.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997114       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:09.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997117       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:09.4	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997119       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:09.5	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997121       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:09.6	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997123       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:09.7	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997126       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:0a.0	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997128       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:0a.1	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997130       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:0a.2	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997133       1 auxNetDeviceProvider.go:84] auxnetdevice AddTargetDevices(): device found: 0000:42:0a.3	02          	Mellanox Technolo...	ConnectX Family mlx5Gen Virtual Function
I1128 02:40:11.997136       1 main.go:68] Initializing resource servers
I1128 02:40:11.997142       1 manager.go:117] number of config: 1
I1128 02:40:11.997153       1 manager.go:121] Creating new ResourcePool: mlnx_sriov_cx7
I1128 02:40:11.997157       1 manager.go:122] DeviceType: netDevice
I1128 02:40:12.116627       1 manager.go:138] initServers(): selector index 0 will register 0 devices
I1128 02:40:12.116690       1 manager.go:142] no devices in device pool, skipping creating resource server for mlnx_sriov_cx7
I1128 02:40:12.116695       1 main.go:74] Starting all servers...
I1128 02:40:12.116701       1 main.go:79] All servers started.
I1128 02:40:12.116704       1 main.go:80] Listening for term signals

What did you expect to happen?

I expected to be able to see mlnx_sriov_cx7 resources available

kubectl get node node1 -o json | jq '.status.allocatable'
{
  "cpu": "384",
  "ephemeral-storage": "423821938396",
  "hugepages-1Gi": "64Gi",
  "hugepages-2Mi": "256Mi",
  "memory": "1516845864Ki",
  "pods": "110"
}

What are the minimal steps needed to reproduce the bug?

I suspect that the issue relates to this line in the log

WARNING: failed to read int from file: open /sys/devices/system/node/node0/cpu0/online: no such file or directory

Anything else we need to know?

I am using tuned to isolate CPU cores

[main]
summary=Profile to isolate CPU cores 0-23 and 279-287 on NUMA node 0

[cpu]
isolated_cores=0-23,279-287

[bootloader]
# Add kernel parameters to isolate cores 0-23 and 279-287, disable RCU callbacks, and remove scheduling overhead
cmdline_append=isolcpus=0-23,279-287 nohz_full=0-23,279-287 rcu_nocbs=0-23,279-287 amd_iommu=on iommu=force pcie_acs_override=downstream,multifunction vfio-pci.ids= selinux=0

[scripts]
start=swapoff -a
stop=swapon -a

Component Versions

Please fill in the below table with the version numbers of components used.

Component Version
SR-IOV Network Device Plugin
SR-IOV CNI Plugin
Multus
Kubernetes
OS

Config Files

Config file locations may be config dependent.

Device pool config file location (Try '/etc/pcidp/config.json')
Multus config (Try '/etc/cni/multus/net.d')
CNI config (Try '/etc/cni/net.d/')
Kubernetes deployment type ( Bare Metal, Kubeadm etc.)
Kubeconfig file
SR-IOV Network Custom Resource Definition

Logs

SR-IOV Network Device Plugin Logs (use kubectl logs $PODNAME)
Multus logs (If enabled. Try '/var/log/multus.log' )
Kubelet logs (journalctl -u kubelet)
@rollandf
Copy link
Contributor

Can you try "rootDevices" instead of "pciAddresses".
"pciAddresses" means only the devices that has these specific PCI address.

@gavmckee80
Copy link
Author

@rollandf I tried using the pfNames as follows

{
  "resourceList": [
    {
      "resourceName": "asap2_vf",
      "resourcePrefix": "nvidia.com",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "devices": [
          "101e"
        ],
        "drivers": [
                "mlx5_core"
        ],
        "pfNames": [
          "ens1f0npf0vf#0-23","ens1f1npf1vf#0-23"
        ]
      }
    },
    {
      "resourceName": "asap2_vfio",
      "resourcePrefix": "nvidia.com",
      "selectors": {
        "vendors": [
          "15b3"
        ],
        "devices": [
          "101e"
        ],
        "drivers": [
          "vfio-pci","mlx5_core"
        ],
        "pfNames": [
          "ens1f0v#0-23","ens1f1v#0-23"
        ]
      }
    }
  ]
}

As a test I added both vfio and mlx5_core as drivers under the VF , all the VF on one phy interface are vfio bound the other are still mlx5_core. Even with that I still don't get any devices.
A log attached and lspci output also

sriovdp.INFO.log
lspci-output.txt

@gavmckee80
Copy link
Author

Just a quick follow up , when I remove the pfNames , I then see the resources being populated.

kubectl get node vaeq-cu2a-r113-lab-staging-hv-05.vaeq-lab-staging.infra.cx -o json | jq '.status.allocatable'
{
  "cpu": "384",
  "devices.kubevirt.io/kvm": "1k",
  "devices.kubevirt.io/tun": "1k",
  "devices.kubevirt.io/vhost-net": "1k",
  "ephemeral-storage": "423821938396",
  "hugepages-1Gi": "64Gi",
  "hugepages-2Mi": "256Mi",
  "memory": "1516845912Ki",
  "nvidia.com/asap2_vf": "24",
  "nvidia.com/asap2_vfio": "24",
  "pods": "110"
}

Is it something in match pattern , you can see from the attached logs and output that the pfNames seem to be correct.

@gavmckee80
Copy link
Author

@rollandf any idea why when I use the patterns here #609 (comment) that it won't discover the devices ?

@rollandf
Copy link
Contributor

rollandf commented Dec 2, 2024

I would try without the 'vf' for NVIDIA NICs:

        "pfNames": [
          "ens1f0npf0#0-23","ens1f1npf1#0-23"
        ]

@rollandf
Copy link
Contributor

rollandf commented Dec 9, 2024

@gavmckee80 can we close this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants