Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

USB proxy: support 3.2 Gen 2×2 (20Gbps) and USB4 (40Gbps) #9587

Open
mzpqnxow opened this issue Nov 18, 2024 · 10 comments · May be fixed by QubesOS/qubes-app-linux-usb-proxy#46
Open

USB proxy: support 3.2 Gen 2×2 (20Gbps) and USB4 (40Gbps) #9587

mzpqnxow opened this issue Nov 18, 2024 · 10 comments · May be fixed by QubesOS/qubes-app-linux-usb-proxy#46
Labels
C: usb proxy P: minor Priority: minor. The lowest priority, below "default." pr submitted A pull request has been submitted for this issue. T: enhancement Type: enhancement. A new feature that does not yet exist or improvement of existing functionality.

Comments

@mzpqnxow
Copy link

NOTE: This is practically the same as #6002 as far as I can tell, except it's 20000 for USB 3.2 Gen 2x2 and 40000 for USB4. I have not tested USB4 but am happy to do so if/when this is fixed. I'm encountering this issue using Highpoint RM110, which is a USB4 NVME enclosure that can operate on any USB speed (5000, 10000, 20000, 40000)

NOTE: This should not be considered a blocker or high priority issue as I am able to use qvm-block rather than relying on usb proxy - something I prefer to do with storage devices anyway. I only tried usb proxy (and encountered this issue) because I wanted to compare the performance of each (both the throughput as well as the CPU usage)

ONE MORE NOTE: I made a very deliberate effort to ensure that the peripheral and the controller would not use USB tunneled inside Thunderbolt, which is a common way for storage peripherals to support both Thunderbolt and USB4 - even though pure USB can operate at both 20Gbps and 40Gbps. It's a somewhat confusing topic but I'm mentioning it to avoid any misinterpretation about the protocol being USB or Thunderbolt. The peripheral only supports "pure" USB4 and to be safe, I also have disabled Thunderbolt in UEFI, for all sorts of reasons. So, yes, this is USB protocol, there is no Thunderbolt involved here. I prefer my PCI bridges inside my computer 😄

The problem you're addressing (if any)

I would like to use a 20Gbps (and eventually, 40Gbps) USB device on QubesOS using usb proxy

When I try to attach to an AppVM via usb proxy, I get the following in dom0:

Nov 18 18:18:45 personal qubes.USBAttach-dom0[2503]: Invalid speed "20000" received. Expected "1.5", "12", "480", "53.3-480", "5000", "10000".  If the remote side sent nothing, this could mean    - the device is invalid or unplugged   - the VM crashed   - qubes-usb-proxy is not installe>

Attaching to the same AppVM using block attach works wonderfully and the performance is fantastic

The solution you'd like

I would like for USB proxy attachment of a 20Gbps or 40Gbps (USB4) device to work

The value to a user, and who that user might be

In theory, any user making use of external storage utilizing USB Gen 2x2 or USB4 for high-performance peripherals

In practice, probably 90% of such users are using an enclosure of some sort, and are therefore not blocked (pardon the pun) as they can use qvm-block

Only users with USB3.2 Gen 2x2 / USB4 peripherals that do not present as block device would actually benefit from supporting these higher speeds. I'm not familiar with the applications of such high speeds outside of storage, but I can guess that perhaps ultra high definition video capture might operate over 20Gbps or 40Gbps link speeds. Pretty niche...

Completion criteria checklist

@mzpqnxow mzpqnxow added P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. T: enhancement Type: enhancement. A new feature that does not yet exist or improvement of existing functionality. labels Nov 18, 2024
marmarek added a commit to marmarek/qubes-app-linux-usb-proxy that referenced this issue Nov 19, 2024
USBIP in Linux 6.12 accepts USB_SPEED_SUPER_PLUS now (see
bb548c1654db4a6f1a42657ed9068237e0c044b5 in Linux). Configure USB 3.1
device this way if kernel is new enough.

QubesOS/qubes-issues#9587
marmarek added a commit to marmarek/qubes-app-linux-usb-proxy that referenced this issue Nov 19, 2024
Attach it as USB 3.1, until USBIP gets support for higher speed.

Fixes QubesOS/qubes-issues#9587
@marmarek
Copy link
Member

Can you check if it works if you modify /usr/lib/qubes/usb-import in the target qube? See PR I just opened, but you can probably use simplified version with just 20000) speed=5 ;;. Support for speed=6 (USB 3.1) is in Linux 6.12 for which we don't have a package yet (we usually wait until .2 or .3 with the new stable branch).

@mzpqnxow
Copy link
Author

From a cosmetic point of view, that resolves it - however, it doesn't quite work:

[ 6189.961320] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[ 6189.961410] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 2
[ 6189.961437] vhci_hcd: created sysfs vhci_hcd.0
[ 6189.961478] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.06
[ 6189.961514] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 6189.961537] usb usb2: Product: USB/IP Virtual Host Controller
[ 6189.961556] usb usb2: Manufacturer: Linux 6.6.54-1.qubes.fc37.x86_64 vhci_hcd
[ 6189.961583] usb usb2: SerialNumber: vhci_hcd.0
[ 6189.961730] hub 2-0:1.0: USB hub found
[ 6189.961749] hub 2-0:1.0: 8 ports detected
[ 6189.961872] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[ 6189.961998] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 3
[ 6189.962027] usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
[ 6189.962059] usb usb3: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.06
[ 6189.962077] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[ 6189.962094] usb usb3: Product: USB/IP Virtual Host Controller
[ 6189.962108] usb usb3: Manufacturer: Linux 6.6.54-1.qubes.fc37.x86_64 vhci_hcd
[ 6189.962126] usb usb3: SerialNumber: vhci_hcd.0
[ 6189.962210] hub 3-0:1.0: USB hub found
[ 6189.962225] hub 3-0:1.0: 8 ports detected
[ 6190.205034] vhci_hcd vhci_hcd.0: pdev(0) rhport(0) sockfd(0)
[ 6190.205053] vhci_hcd vhci_hcd.0: devid(131074) speed(5) speed_str(super-speed)
[ 6190.205082] vhci_hcd vhci_hcd.0: Device attached
[ 6190.446499] usb 3-1: SetAddress Request (2) to port 0
[ 6190.446516] usb 3-1: new SuperSpeed USB device number 2 using vhci_hcd
[ 6190.464593] usb 3-1: New USB device found, idVendor=1103, idProduct=0110, bcdDevice= 1.00
[ 6190.464613] usb 3-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[ 6190.464629] usb 3-1: Product: RM110
[ 6190.464638] usb 3-1: Manufacturer: HighPoint
[ 6190.464649] usb 3-1: SerialNumber: 2219AFF000045
[ 6190.480726] usb 3-1: USB controller vhci_hcd.0 does not support streams, which are required by the UAS driver.
[ 6190.480751] usb 3-1: Please try an other USB controller if you wish to use UAS.
[ 6190.480768] usb-storage 3-1:1.0: USB Mass Storage device detected
[ 6190.480989] scsi host2: usb-storage 3-1:1.0
[ 6190.481105] usbcore: registered new interface driver usb-storage
[ 6190.482323] usbcore: registered new interface driver uas
[ 6191.658587] usb 3-1: SetAddress Request (2) to port 0
[ 6191.658626] usb 3-1: reset SuperSpeed USB device number 2 using vhci_hcd
[ 6191.805570] usb 3-1: SetAddress Request (2) to port 0
[ 6191.805619] usb 3-1: reset SuperSpeed USB device number 2 using vhci_hcd
[ 6191.951539] usb 3-1: SetAddress Request (2) to port 0
[ 6191.951558] usb 3-1: reset SuperSpeed USB device number 2 using vhci_hcd
[ 6192.097568] usb 3-1: SetAddress Request (2) to port 0
[ 6192.097615] usb 3-1: reset SuperSpeed USB device number 2 using vhci_hcd

@mzpqnxow
Copy link
Author

If you're curious, this is what it looks like when plugged in to sys-usb, before passing it via usb proxy:

[17438.293614] usb 2-1: new SuperSpeed Plus Gen 2x2 USB device number 3 using xhci_hcd
[17438.310038] usb 2-1: New USB device found, idVendor=1103, idProduct=0110, bcdDevice= 1.00
[17438.310063] usb 2-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[17438.310079] usb 2-1: Product: RM110
[17438.310089] usb 2-1: Manufacturer: HighPoint
[17438.310101] usb 2-1: SerialNumber: 2219AFF000045
[17438.333345] scsi host2: uas
[17439.248270] scsi 2:0:0:0: Direct-Access     HPT      RocketMaet RM110 0    PQ: 0 ANSI: 6
[17441.139169] sd 2:0:0:0: Attached scsi generic sg0 type 0
[17441.543455] sd 2:0:0:0: [sda] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[17441.543601] sd 2:0:0:0: [sda] Write Protect is off
[17441.543637] sd 2:0:0:0: [sda] Mode Sense: 43 00 00 00
[17441.543778] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[17441.543967] xhci_hcd 0000:00:06.0: bad transfer trb length 112 in event trb
[17441.567921] sd 2:0:0:0: [sda] Preferred minimum I/O size 512 bytes
[17441.567938] sd 2:0:0:0: [sda] Optimal transfer size 2097152 bytes
[17441.569851]  sda: sda1
[17441.569979] sd 2:0:0:0: [sda] Attached SCSI disk

The line:

[17441.543967] xhci_hcd 0000:00:06.0: bad transfer trb length 112 in event trb

... is somewhat curious. But it does work fine, in sys-usb and when passed via qvm-block

@mzpqnxow
Copy link
Author

mzpqnxow commented Nov 19, 2024

Regarding the power errors - it’s possible that I used procfs/sysfs to disable FLR on the controller associated with this device. I’m not sure that matters as it works fine in sys-usb, but seemed prudent to mention

@marmarek
Copy link
Member

This might be something else, what happens if you try qvm-service --enable sys-usb usb-reset-on-attach ?

@marmarek
Copy link
Member

Hmm, this looks very similar to #9546 . Can you try kernel 6.6.60 (or 6.6.63) that should have it fixed already?

@mzpqnxow
Copy link
Author

Hmm, this looks very similar to #9546 . Can you try kernel 6.6.60 (or 6.6.63) that should have it fixed already?

Ahh interesting, yes, I'm AFK but will try to give it a shot as soon as I can

Thanks!

@andrewdavidwong andrewdavidwong added C: usb proxy P: minor Priority: minor. The lowest priority, below "default." and removed P: default Priority: default. Default priority for new issues, to be replaced given sufficient information. labels Nov 29, 2024
@andrewdavidwong andrewdavidwong changed the title usb proxy: (low priority) support 3.2 Gen 2×2 (20Gbps) and USB4 (40Gbps) USB proxy: support 3.2 Gen 2×2 (20Gbps) and USB4 (40Gbps) Nov 29, 2024
@mzpqnxow
Copy link
Author

mzpqnxow commented Dec 15, 2024

Tested with reasonable success using all of your suggestions:

  • kernel 6.6.63-1.qubes.fc37.x86_64 with the changes in /usr/lib/qubes/usb-import to not bail on 2x2 speed:
  • with 20000) speed=5 ;; # Super Speed Plus (USB 3.1); Announce as USB 3.0 until USBIP get support into usb-import in target appVM
  • with usb-reset-on-attach set on sys-usb appVM

Those allow the block devices to be seen and the partitions to be probed, though there are some complaints in the target appVM - included below in case they're of interest:

[  187.098643] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[  187.098688] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 2
[  187.098697] vhci_hcd: created sysfs vhci_hcd.0
[  187.098717] usb usb2: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.06
[  187.098724] usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[  187.098730] usb usb2: Product: USB/IP Virtual Host Controller
[  187.098735] usb usb2: Manufacturer: Linux 6.6.63-1.qubes.fc37.x86_64 vhci_hcd
[  187.098740] usb usb2: SerialNumber: vhci_hcd.0
[  187.098786] hub 2-0:1.0: USB hub found
[  187.098793] hub 2-0:1.0: 8 ports detected
[  187.098850] vhci_hcd vhci_hcd.0: USB/IP Virtual Host Controller
[  187.098886] vhci_hcd vhci_hcd.0: new USB bus registered, assigned bus number 3
[  187.098897] usb usb3: We don't know the algorithms for LPM for this host, disabling LPM.
[  187.098910] usb usb3: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.06
[  187.098917] usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[  187.098922] usb usb3: Product: USB/IP Virtual Host Controller
[  187.098926] usb usb3: Manufacturer: Linux 6.6.63-1.qubes.fc37.x86_64 vhci_hcd
[  187.098932] usb usb3: SerialNumber: vhci_hcd.0
[  187.098974] hub 3-0:1.0: USB hub found
[  187.098979] hub 3-0:1.0: 8 ports detected
[  187.446176] vhci_hcd vhci_hcd.0: pdev(0) rhport(0) sockfd(0)
[  187.446198] vhci_hcd vhci_hcd.0: devid(458754) speed(5) speed_str(super-speed)
[  187.446234] vhci_hcd vhci_hcd.0: Device attached
[  187.684196] usb 3-1: SetAddress Request (2) to port 0
[  187.684210] usb 3-1: new SuperSpeed USB device number 2 using vhci_hcd
[  187.699198] usb 3-1: New USB device found, idVendor=04e8, idProduct=61fd, bcdDevice= 1.00
[  187.699212] usb 3-1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
[  187.699217] usb 3-1: Product: PSSD T9
[  187.699221] usb 3-1: Manufacturer: Samsung
[  187.699224] usb 3-1: SerialNumber: S123456789
[  187.715494] usb 3-1: USB controller vhci_hcd.0 does not support streams, which are required by the UAS driver.
[  187.715507] usb 3-1: Please try an other USB controller if you wish to use UAS.
[  187.715512] usb-storage 3-1:1.0: USB Mass Storage device detected
[  187.715649] scsi host2: usb-storage 3-1:1.0
[  187.715698] usbcore: registered new interface driver usb-storage
[  187.716468] usbcore: registered new interface driver uas
[  188.772659] scsi 2:0:0:0: Direct-Access     Samsung  PSSD T9          0    PQ: 0 ANSI: 6
[  188.772955] sd 2:0:0:0: Attached scsi generic sg0 type 0
[  188.773463] sd 2:0:0:0: [sda] Very big device. Trying to use READ CAPACITY(16).
[  188.774094] sd 2:0:0:0: [sda] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[  188.774994] sd 2:0:0:0: [sda] Write Protect is off
[  188.775000] sd 2:0:0:0: [sda] Mode Sense: 43 00 00 00
[  188.775860] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  188.787102]  sda: sda1 sda2
[  188.787200] sd 2:0:0:0: [sda] Attached SCSI disk

I was curious what really did the trick and noticed that enabling usb-reset-on-attach is a critical part of it. Without that, it fails. I don't understand USB well enough to say if that's a device "quirk" or a USB3.1 gen 2x2 (or whatever the proper name is for 20Gbps USB3). Unfortunately, the 20Gbps USB devices I have are all the same model, so I can't contribute much in the way of determining that for sure. I'm betting on the former, based on my prior experience with various 10Gbps and 20Gbps USB drives...

Irrelevant comment: I had some concerns about the usb-reset-on-attach fix impacting other devices which may not handle a reset well, but I'm asuming I can ditch the (convenient) VM-wide Qubes solution and use a kernel USB device quirk or udev + the relevant per-device setting in /sys. So not really an issue

Note that I haven't tested the performance; I assume the best throughput I can expect is (10Gbps - usbip overhead) rather than (20Gbps - usbip overhead) until 6.12 is packaged which is fine. I won't bother building myself because attaching as a block device works at 20Gbps already

Thanks for your time on this - hopefully it will prove worthwhile, since it seems it should make it quick to add the support for 20Gbps (and maybe 40Gbps / USB4) as soon as 6.12 is packaged

tl; dr; Suggestions resolved the issue; forced USB reset in the target VM seemed to be critical; thanks, I'll try to ping about (or send a PR for) the speed=6 setting once 6.12 is packaged, to save you a few minutes. If you would like to close this, please do

@marmarek
Copy link
Member

6.12 is in the testing repo already, since yesterday

@andrewdavidwong andrewdavidwong added the pr submitted A pull request has been submitted for this issue. label Dec 15, 2024
@mzpqnxow
Copy link
Author

Fantastic, I will give it a try when it hits stable. I'm a cautious fellow with my data, which means waiting more than I probably need to when it comes to kernel updates 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C: usb proxy P: minor Priority: minor. The lowest priority, below "default." pr submitted A pull request has been submitted for this issue. T: enhancement Type: enhancement. A new feature that does not yet exist or improvement of existing functionality.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants