PC freezes/crashes after launching two Realsense D435I cameras with ROS2

[norakon]

---

Required Info
Camera Model	D400
Firmware Version	5.16.0.1
Operating System & Version	Docker Container with Ubuntu 22
Kernel Version (Linux Only)	6.8
Platform	PC
SDK Version	v4.55.1
Language	ROS2 / C++
Segment	Robot

Issue Description

We are running two Realsense D435I cameras inside a Docker container on our mobile robotic platform. After launching the rs_multi_camera_launch.py , together with the motor controller (connected via USB), it takes between 5 - 60 minutes until the PC freezes entirely, and needs to be power-cycled to restart. Just as described here, #12016, we, too, suspect that it is related to the Realsense Cameras because the PC does not crash when the cameras are not turned on. Also, using cameras from other brands, the PC does not freeze.
The cameras are directly connected to the PC's USB ports.

--

We have tried two different PCs from different brands.

1. The first PC has the following specifications:

   Kernel: 6.8.0-50-generic
   OS: Ubuntu 22.04
   CPU: 13th Gen Intel Core i7-13700
   GPU: NVIDIA T1000 8GB
   RAM: 32GB

2. The second PC has the following specifications:

   Kernel: 6.8.0-50-generic
   OS: Ubuntu 22.04
   CPU: 13th Gen Intel Core i7-13700TE
   GPU: NVIDIA T1000 8GB
   RAM: 32GB

We, too, have tried two different methods to install the SDK.

• The first was building the librealsense2 SDK from source, as described here: https://github.com/IntelRealSense/librealsense/blob/development/doc/installation.md, which completed successfully.
• And the second was to install it through the package manager, as described here: https://github.com/IntelRealSense/librealsense/blob/development/doc/distribution_linux.md#installing-the-packages, which also completed successfully

We have also tried older kernel versions with the respective SDK:

• 5.19
• 6.8
• 6.12

--> Regardless, in all the cases (different PCs, different SDK installation methods, different kernels), both PCs froze a few minutes after launching the realsense cameras.

Some surprising findings:

• Running cameras from another brand with the motor controller, did not cause the PC to freeze, so far.
• Running only one realsense camera did not cause a PC freeze, so far.
• Running two realsense cameras without the motor controller, did not cause the PC freeze, so far.
• Running the two realsense cameras with the motor controller, on an older PC with an Intel I9-10900, but same kernel, did not cause the PC freeze, so far.

[MartyG-RealSense]

Hi @norakon If there is not a freeze if one RealSense camera + the motor controller is used on the 13th Generation PCs, or if two cameras are on the 10th Gen PC, you could check for the possibility of whether the 13th Gen PCs are providing insufficient USB power for reliable operation of two RealSense cameras and the controller. Although USB is a standard, not all USB ports are equal and some PCs have more reliable ports than others.

As a first step in this support discussion, I recommend testing on the 13th Gen PCs for whether USB port power-saving is kicking in and causing insufficient power for two RealSense cameras -and- a USB motor controller simultaneously by using the Ubuntu terminal command below to disable USB power-saving so that port power saving does not kick in and cause a camera's operation to fail.

echo on | sudo tee /sys/bus/usb/devices/*/power/level > /dev/null

[norakon]

Hi @MartyG-RealSense, thank you for your quick response!
We tried your command to ensure that the USB always stays powered on, but it did not help. Within a few minutes the PC (with 13th Generation Intel Processor) froze again.

[MartyG-RealSense]

Given that you can run two RealSense cameras on 13th Gen when the motor controller is not active and not experience the PC freeze, do you have the option of testing a different motor controller?

[norakon]

Yes, we have tried a different motor controller from a different brand, that comes with it's own drivers. The only similarity is that it is also connected via USB. But also there, both PCs freeze.

[norakon]

Today, we tested the two cameras and the motor controller on Windows. We tried to run the two realsense cameras with the realsense viewer, and the motor controller with its corresponding windows application. Also there, the PC froze.
This means, the problem is not related to Linux or our ROS2 setup.

[MartyG-RealSense]

Instead of using rs_multi_camera_launch.py to launch both cameras simultaneously, does it make a difference if you create two terminals and launch the cameras individually (one for each terminal) with the rs_launch.py launch file?

Terminal 1
ros2 launch realsense2_camera rs_launch.py camera_namespace:=robot1 camera_name:=D435i_1

Terminal 2
ros2 launch realsense2_camera rs_launch.py camera_namespace:=robot1 camera_name:=D435i_2

[norakon]

No, separating the launches unfortunately did not help either.

[MartyG-RealSense]

Another user at IntelRealSense/realsense-ros#3285 reported yesterday that they were having the same problems as you in ROS with 2 RealSense cameras + a controller not working with a Gen 13 PC but working with Gen 10.

[MartyG-RealSense]

Hi @norakon Do you have an update about this case that you can provide, please? Thanks!

[norakon]

We tested using an externally powered USB hub, where we connected both cameras and the motor controller. The PCs did not freeze this way. However, there were some network delays that caused another set of problems. So we moved away from this solution, again.
Our current solution is not using the USB interface for the motor controller, but to connect it differently. That seems to work best, at the current stage.

[MartyG-RealSense]

I'm pleased to hear that you found a workaround for the motor controller issue, @norakon

[MartyG-RealSense]

Hi @norakon Do you require further assistance with this case, please? Thanks!

[norakon]

We're about to switch the processor in one of the PCs to see if the processor generation causes the issue. I would like to leave this issue open until then, so that we can share our findings.

[MartyG-RealSense]

Thanks very much for the update. It's no problem at all to keep the issue open until you have switched the processor.

[norakon]

Hi @MartyG-RealSense, changing the processor to a 12th generation processor did not help. We can conclude our PC freezing issue is not related to the reported CPU instabilities.
I think we can close this thread, now.

[MartyG-RealSense]

Okay, thanks very much for the update. As you suggest, I will close the thread. Thanks again!