T265 and D455 not enumerating after USB power reset

[3zuli]

Required Info
Camera Model	T265, D455
Firmware Version	T265: 0.2.0.951, D455: 05.12.12.100
Operating System & Version	Ubuntu 18
Kernel Version (Linux Only)	Jetpack 4.5 (L4T 32.5.1)
Platform	NVIDIA Jetson Xavier NX
SDK Version	2.43.0
ROS Version	ROS Melodic, realsense-ros 2.2.20

Issue Description

We are trying to simultaneously use the T265 camera for odometry and the D455 for depth / point cloud measurements. We have the problem described in #4681, where the T265 does not enumerate properly during system boot. The suggested solution was to reset the root USB hub power using uhubctl, which we are doing before we launch any of our ROS nodes. Previously we were using the D435 camera, which worked fine and was not affected by this workaround.

The problem occurred now when we wanted to upgrade to the D455 camera. It behaves exactly opposite of the T265: it enumerates fine during system boot, but fails to re-enumerate after the USB power reset.

After initial system boot the D455 works, the T265 does not:

$ rs-enumerate-devices -s
 20/08 08:20:47,399 ERROR [548517545024] (tm-boot.h:39) Error booting T265
 20/08 08:20:50,451 ERROR [548517545024] (tm-boot.h:39) Error booting T265
Device Name                   Serial Number       Firmware Version    
Intel RealSense D455          035122250211        05.12.12.100        

$ sudo uhubctl
Current status for hub 2-3 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports, ppps]
  Port 1: 02a0 power 5gbps Rx.Detect
  Port 2: 02a0 power 5gbps Rx.Detect
  Port 3: 0203 power 5gbps U0 enable connect [8086:0b5c Intel(R) RealSense(TM) Depth Camera 455 Intel(R) RealSense(TM) Depth Camera 455 026123052589]
  Port 4: 02a0 power 5gbps Rx.Detect
Current status for hub 1-2 [0bda:5411 Generic 4-Port USB 2.1 Hub, USB 2.10, 4 ports, ppps]
  Port 1: 0100 power
  Port 2: 0503 power highspeed enable connect [03e7:2150 Movidius Ltd. Movidius MA2X5X 03e72150]
  Port 3: 0100 power
  Port 4: 0100 power

To fix the T265, we perform the USB power reset:

$ sudo uhubctl -l 1-2 -a cycle -d 5
Current status for hub 2-3 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports, ppps]
  Port 1: 02a0 power 5gbps Rx.Detect
  Port 2: 02a0 power 5gbps Rx.Detect
  Port 3: 0203 power 5gbps U0 enable connect [8086:0b5c Intel(R) RealSense(TM) Depth Camera 455 Intel(R) RealSense(TM) Depth Camera 455 026123052589]
  Port 4: 02a0 power 5gbps Rx.Detect
Sent power off request
New status for hub 2-3 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports, ppps]
  Port 1: 00a0 off
  Port 2: 00a0 off
  Port 3: 00a0 off
  Port 4: 00a0 off
Current status for hub 1-2 [0bda:5411 Generic 4-Port USB 2.1 Hub, USB 2.10, 4 ports, ppps]
  Port 1: 0100 power
  Port 2: 0503 power highspeed enable connect [03e7:2150 Movidius Ltd. Movidius MA2X5X 03e72150]
  Port 3: 0100 power
  Port 4: 0100 power
Sent power off request
New status for hub 1-2 [0bda:5411 Generic 4-Port USB 2.1 Hub, USB 2.10, 4 ports, ppps]
  Port 1: 0000 off
  Port 2: 0000 off
  Port 3: 0008 off
  Port 4: 0008 off
Current status for hub 2-3 [0bda:0411 Generic 4-Port USB 3.1 Hub, USB 3.10, 4 ports, ppps]
  Port 1: 00a8 off
  Port 2: 00a8 off
  Port 3: 00a8 off
  Port 4: 00a8 off
Sent power on request

dmesg output during reset:

[  121.693642] usb 1-2.2: USB disconnect, device number 4
[  121.815907] usb 1-2: reset high-speed USB device number 2 using tegra-xusb
[  121.991395] usb 1-2: usb_suspend_both: status 0
[  122.503385] usb 1-2.2: new high-speed USB device number 5 using tegra-xusb
[  122.524523] usb 1-2.2: New USB device found, idVendor=03e7, idProduct=2150
[  122.524542] usb 1-2.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  122.524551] usb 1-2.2: Product: Movidius MA2X5X
[  122.524557] usb 1-2.2: Manufacturer: Movidius Ltd.
[  122.524563] usb 1-2.2: SerialNumber: 03e72150

At this point the T265 enumerates, but the D455 does not enumerate:

$ rs-enumerate-devices -s
Could not create device - set_xu(ctrl=1) failed! Last Error: Input/output error . Check SDK logs for details
Device Name                   Serial Number       Firmware Version    
Intel RealSense T265          109422110706        0.2.0.951

dmesg during rs-enumerate-devices:

[  131.684303] usb 1-2.2: USB disconnect, device number 5
[  131.875371] usb 1-2: usb_suspend_both: status 0
[  131.891812] usb 2-3.2: new SuperSpeed USB device number 4 using tegra-xusb
[  131.913175] usb 2-3.2: New USB device found, idVendor=8087, idProduct=0b37
[  131.913223] usb 2-3.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  131.913232] usb 2-3.2: Product: Intel(R) RealSense(TM) Tracking Camera T265
[  131.913238] usb 2-3.2: Manufacturer: Intel(R) Corporation
[  131.913291] usb 2-3.2: SerialNumber: 109422110706
[  133.479181] uvcvideo: Failed to query (SET_CUR) UVC control 1 on unit 3: -71 (exp. 1024).
[  133.530607] uvcvideo: Failed to query (SET_CUR) UVC control 1 on unit 3: -71 (exp. 1024).
[  133.582013] uvcvideo: Failed to query (SET_CUR) UVC control 1 on unit 3: -71 (exp. 1024).
# Many more errors like this...
[  138.489756] uvcvideo: Failed to query (SET_CUR) UVC control 1 on unit 3: -71 (exp. 1024).
[  138.541441] uvcvideo: Failed to query (SET_CUR) UVC control 1 on unit 3: -71 (exp. 1024).
[  138.593152] uvcvideo: Failed to query (SET_CUR) UVC control 1 on unit 3: -71 (exp. 1024).
# The errors stop when rs-enumerate-devices exits

This error has been reported in the past (#5302, #7060), however I wasn't able to find any definite solution to it.

If I perform another USB reset as an attempt to fix the D455, both cameras no longer enumerate:

$ rs-enumerate-devices -s
Could not create device - set_xu(ctrl=1) failed! Last Error: Input/output error . Check SDK logs for details
Could not create device - Unable to communicate with device . Check SDK logs for details
Device Name                   Serial Number       Firmware Version    
 20/08 08:49:34,671 ERROR [548268663872] (tm-device.cpp:2038) Bulk request error RS2_USB_STATUS_IO

At this point the only way to fix it is to manually re-connect both cameras. In this case they both enumerate correctly. If I power the system off and on, it goes back to the state where the D455 works and T265 does not enumerate.

[3zuli]

I think I found the issue. It seems that our root USB hub was somehow ending up in an inconsistent state after the power cycle, which interfered with the cameras and caused them not to enumerate. The solution was to use uhubctl with the argument -R - reset hub after each power-on action, causing all devices to reassociate.

sudo uhubctl -l 1-2 -a cycle -d 5 -R

[manyxu]

can not work on jetson-xavier :-(
after restart linux system, can not find realsense t265

[fallow24]

This command worked for me on a Raspberry Pi 4B using a single T265. The issue was that the T265 would not be able to boot after a system reboot.

sudo uhubctl -l 1-1 -a cycle -d 5 -R

After that, booting the T265 works. However, I must only run this command once after the system reboot, and NOT before every T265 launch. Otherwise the T265 won't boot anymore.