Camera does not (re-)start after "failed to set power state" exception

[stwirth]

Very similar to #1734 I am seeing occasional failures of bringing up two RealSense D410 cameras at the same time. This can be reproduced in about 20-40% of all power cycles.

Some system information:
Ubuntu 20.04
ROS noetic
realsense2_camera version: 2.3.2

$ uname -a
Linux XXXX 5.11.0-46-generic #51~20.04.1-Ubuntu SMP Fri Jan 7 06:51:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

$ rs-enumerate-devices --version
rs-enumerate-devices  version: 2.50.0

$ rs-enumerate-devices -s
Device Name                   Serial Number       Firmware Version    
Intel RealSense D410          747512020688        05.13.00.50         
Intel RealSense D410          747512020684        05.13.00.50

When the failure happens, the corresponding node logs:

1647003638.944730203 ERROR /camera_down_orig/nodelet_manager [/root/upstream_ws/src/realsense-ros-release-release-noetic-realsense2_camera-2.3.2-1/src/base_realsense_node.cpp:952(BaseRealSenseNode::setupDevice)] [topics: /rosout, /camera_down_orig/nodelet_manager/bond, /tf_static, /tf] An exception has been thrown: failed to set power state
1647003638.944828869 ERROR /camera_down_orig/nodelet_manager [/root/upstream_ws/src/realsense-ros-release-release-noetic-realsense2_camera-2.3.2-1/src/realsense_node_factory.cpp:394(RealSenseNodeFactory::StartDevice)] [topics: /rosout, /camera_down_orig/nodelet_manager/bond, /tf_static, /tf] Exception: failed to set power state

which are emitted in these lines:

realsense-ros/realsense2_camera/src/base_realsense_node.cpp

Line 952 in f400d68

ROS_ERROR_STREAM("An exception has been thrown: " << ex.what());

realsense-ros/realsense2_camera/src/realsense_node_factory.cpp

Line 394 in f400d68

ROS_ERROR_STREAM("Exception: " << e.what());

Afterwards, the node is still pingable with rosnode ping but no images are published.

A call of the reset service rosservice call /camera_down_orig/driver/reset successfully brings the camera back to a working state.
So I think independent of what the cause for the "failed to set power state" exception might be, the bringup logic in RealSenseNodeFactory::initialize does not retry enough.

Maybe _device should be reset after L394 in realsense_node_factory.cpp?

[MartyG-RealSense]

Hi @stwirth A few RealSense users who have recently had stability issues have found that their problem was resolved if they used kernel 5.11.0-34-generic

[stwirth]

I will test with that kernel and report results. Independent of the results, I think the re-connection logic should be modified to retry here.

[stwirth]

Tested again with kernel 5.11.0-34-generic. The results are similar to what I saw with 5.11.0-46-generic. In 20 power cycles, the exception occurred 7 times resulting in one of the two cameras not streaming images.

[MartyG-RealSense]

How are you launching the cameras, please? Are you using a roslaunch instruction, and if you are, have you tried including initial_reset:=true in the roslaunch instruction to reset the camera at launch?

[stwirth]

I can reproduce this issue with

roslaunch realsense2_camera rs_multiple_devices.launch serial_no_camera1:=$CAMERA1_ID serial_no_camera2:=$CAMERA2_ID initial_reset:=true

[MartyG-RealSense]

I recall that in past cases with detection involving a pair of cameras, one approach was to create a sleep delay between the cameras being launched instead of all of them being launched simultaneously. #1187 (comment) provides a table of extensive testing of different methods, including a sleep period between launches, that one RealSense ROS user created.

#774 (comment) has an example of launch file code for defining a launch delay for the second camera.

[stwirth]

Here is a script that repeatedly launches two cameras and checks if they are streaming: https://gist.github.com/stwirth/4fcd3462dbca929d008a742b52900e36
With that I can reproduce the issue with this frequency:

*** Camera1 failed 37 times out of 50
*** Camera2 failed 7 times out of 50

The gist also contains the full log of 50 iterations.

[MartyG-RealSense]

Have you tested your project with an RSUSB backend build of librealsense, as described in #1734 (comment) to determine whether bypassing the kernel with RSUSB can resolve your multicam launch problem?

[stwirth]

I'm already using the RSUSB backend. @doronhi enabled that by default for the ROS build: https://github.com/IntelRealSense/librealsense2-release/blob/release/noetic/librealsense2/2.50.0-1/CMake/lrs_options.cmake#L33

[MartyG-RealSense]

If you are using a librealsense build based upon RSUSB then the kernel should not be an influencing factor in the problem, since an RSUSB build of librealsense is not dependent on Linux versions or kernel versions and does not require patching.

Did you build librealsense and the wrapper together from packages using the Method 1 instructions in the wrapper documentation? Whilst RSUSB can make use of multiple cameras, it is not the ideal backend to use for multicam and a non-RSUSB build (native / V4L backend) from source code and patching the kernel is preferable for multicam applications.

Which computer / computing device are you using, please (Nvidia Jetson, Raspberry Pi, laptop / desktop PC, etc)

[stwirth]

We build librealsense2 and realsense2_camera from the source that is contained in the release repositories (https://github.com/IntelRealSense/librealsense2-release and https://github.com/IntelRealSense/realsense-ros-release). Patching the kernel would not an option for us at the moment.

The processor is an AMD Ryzen Embedded V2516.

[MartyG-RealSense]

RealSense users have reported being able to use the camera with AMD Ryzen Threadripper processors in the past, though we do not have confirmation of Ryzen Embedded.

[stwirth]

Thanks @MartyG-RealSense for your effort.
Just to clarify: The cameras work and I can start them sequentially. Adding an artificial delay between launching the first and the second is an effective workaround for this issue.
Just when launching both cameras simultaneously, a race condition occurs that manifests in one driver not retrying to connect after the "Exception: failed to set power state" exception. The state is that both drivers are running but only one is streaming. Calling the reset ROS service of the failing driver makes the second camera start streaming fine.
I think with a small code change (see initial issue description above), this can be resolved.

[MartyG-RealSense]

Thanks very much @stwirth for the positive report about the success that can be achieved by implementing a launch delay period.

If a RealSense user is able to create the proposed code change then they are welcome to submit it to the RealSense ROS development team as a Pull Request (PR) so that it can be considered for merging into the wrapper.

https://github.com/IntelRealSense/realsense-ros/pulls

[MartyG-RealSense]

Hi @stwirth Do you require further assistance with this case, please? Thanks!

[stwirth]

No, I have a workaround and won't work on a "real fix" in the near future. Thanks for the assistance @MartyG-RealSense!

[MartyG-RealSense]

Thanks very much @stwirth for the update!