Vision - systems failure - "overload"

[RodrigoFBernardo]

Hi,

I am starting to work with a vision for object detection. I started by testing using ar_tags detection, using the VISP library for that. So far, everything works perfectly. When I add this node to the remaining nodes of my system, I start to have communication errors; for example, the communication with the robotic arm controller is lost. My system is overloaded with too much information, and it becomes "heavy" subscribing to my RGBD camera topics.

At first, I thought the problem was in the code I wrote, but I tested it with the ar_tag ([http://wiki.ros.org/ar_track_alvar]) code, and it happened the same. I have also changed the frequency of the publishers and subscribers of the vision system node, but the problem persists.

My problem arises when I subscribe to the camera topics. It only works fine if I only use the camera.

Has anyone ever had a similar problem? My system consists of a mobile base and a robotic arm, and I use several nos. However, it only happens with the vision.

Thank you in advance for your help and attention.

[MartyG-RealSense]

Hi @RodrigoCodeBernardo May I first ask the following questions, please.

1. Are you using a RealSense depth camera, and if you are then which RealSense model is it

2. If you are using a RealSense camera with the RealSense ROS wrapper software, could you provide the launch instruction that you are using and a copy of the launch log from the ROS terminal

If you are not using the RealSense ROS wrapper, are you using a RealSense camera as a general camera with the mentioned AR applications?

[RodrigoFBernardo]

Hi,

I use a Intel Realsense 415

roslaunch realsense_ros_camera rs_rgbd.launch
... logging to /home/idmind/.ros/log/3ef8d212-b423-11ec-a6c1-e0d55e204096/roslaunch-idmind-ipcb-32542.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
WARNING: disk usage in log directory [/home/idmind/.ros/log] is over 1GB.
It's recommended that you use the 'rosclean' command.

started roslaunch server http://192.168.1.100:37775/

SUMMARY
========

PARAMETERS
 * /camera/realsense_ros_camera/accel_fps: 1000
 * /camera/realsense_ros_camera/accel_optical_frame_id: camera_accel_opti...
 * /camera/realsense_ros_camera/align_depth: True
 * /camera/realsense_ros_camera/camera_aligned_depth_to_color_frame_id: camera_aligned_de...
 * /camera/realsense_ros_camera/camera_aligned_depth_to_fisheye_frame_id: camera_aligned_de...
 * /camera/realsense_ros_camera/camera_aligned_depth_to_infra1_frame_id: camera_aligned_de...
 * /camera/realsense_ros_camera/camera_aligned_depth_to_infra2_frame_id: camera_aligned_de...
 * /camera/realsense_ros_camera/color_fps: 30
 * /camera/realsense_ros_camera/color_height: 480
 * /camera/realsense_ros_camera/color_optical_frame_id: camera_color_opti...
 * /camera/realsense_ros_camera/color_width: 640
 * /camera/realsense_ros_camera/depth_fps: 30
 * /camera/realsense_ros_camera/depth_height: 480
 * /camera/realsense_ros_camera/depth_optical_frame_id: camera_depth_opti...
 * /camera/realsense_ros_camera/depth_width: 640
 * /camera/realsense_ros_camera/enable_color: True
 * /camera/realsense_ros_camera/enable_depth: True
 * /camera/realsense_ros_camera/enable_fisheye: True
 * /camera/realsense_ros_camera/enable_imu: True
 * /camera/realsense_ros_camera/enable_infra1: True
 * /camera/realsense_ros_camera/enable_infra2: True
 * /camera/realsense_ros_camera/enable_pointcloud: False
 * /camera/realsense_ros_camera/enable_sync: True
 * /camera/realsense_ros_camera/fisheye_fps: 30
 * /camera/realsense_ros_camera/fisheye_height: 480
 * /camera/realsense_ros_camera/fisheye_optical_frame_id: camera_fisheye_op...
 * /camera/realsense_ros_camera/fisheye_width: 640
 * /camera/realsense_ros_camera/gyro_fps: 1000
 * /camera/realsense_ros_camera/gyro_optical_frame_id: camera_gyro_optic...
 * /camera/realsense_ros_camera/infra1_fps: 30
 * /camera/realsense_ros_camera/infra1_height: 480
 * /camera/realsense_ros_camera/infra1_optical_frame_id: camera_infra1_opt...
 * /camera/realsense_ros_camera/infra1_width: 640
 * /camera/realsense_ros_camera/infra2_fps: 30
 * /camera/realsense_ros_camera/infra2_height: 480
 * /camera/realsense_ros_camera/infra2_optical_frame_id: camera_infra2_opt...
 * /camera/realsense_ros_camera/infra2_width: 640
 * /camera/realsense_ros_camera/json_file_path:
 * /camera/realsense_ros_camera/serial_no:
 * /rosdistro: kinetic
 * /rosversion: 1.12.12

NODES
  /camera/
    color_rectify_color (nodelet/nodelet)
    points_xyzrgb_hw_registered (nodelet/nodelet)
    realsense_ros_camera (nodelet/nodelet)
    realsense_ros_camera_manager (nodelet/nodelet)

auto-starting new master
process[master]: started with pid [32552]
ROS_MASTER_URI=http://192.168.1.100:11311/

setting /run_id to 3ef8d212-b423-11ec-a6c1-e0d55e204096
process[rosout-1]: started with pid [32565]
started core service [/rosout]
process[camera/realsense_ros_camera_manager-2]: started with pid [32582]
process[camera/realsense_ros_camera-3]: started with pid [32583]
process[camera/color_rectify_color-4]: started with pid [32584]
process[camera/points_xyzrgb_hw_registered-5]: started with pid [32586]
[ INFO] [1649082401.975383911]: Initializing nodelet with 8 worker threads.
[ INFO] [1649082402.015509223]: RealSense ROS v2.0.2
[ INFO] [1649082402.015538127]: Running with LibRealSense v2.9.1
[ INFO] [1649082402.264535542]: getParameters...
[ INFO] [1649082402.299289394]: setupDevice...
[ INFO] [1649082402.299321874]: JSON file is not provided
[ INFO] [1649082402.299337369]: ROS Node Namespace: camera
[ INFO] [1649082402.299365317]: Device Name: Intel RealSense 415
[ INFO] [1649082402.299378886]: Device Serial No: 746112061699
[ INFO] [1649082402.299392366]: Device FW version: 05.08.15.00
[ INFO] [1649082402.299402659]: Device Product ID: 0AD3
[ INFO] [1649082402.299413189]: Enable PointCloud: Off
[ INFO] [1649082402.299424046]: Align Depth: On
[ INFO] [1649082402.299438451]: Sync Mode: On
[ INFO] [1649082402.299506636]: Device Sensors:
[ INFO] [1649082402.299533223]: Stereo Module was found.
[ INFO] [1649082402.299548460]: RGB Camera was found.
[ INFO] [1649082402.299578703]: (Fisheye, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1649082402.299592218]: (Gyro, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1649082402.299605030]: (Accel, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1649082402.299621384]: setupPublishers...
[ INFO] [1649082402.447447895]: setupStreams...
[ INFO] [1649082402.452699517]: depth stream is enabled - width: 640, height: 480, fps: 30
[ WARN] [1649082402.452815818]: Given stream configuration is not supported by the device!  Stream: Infrared, Stream Index: 1, Format: Y8, Width: 640, Height: 480, FPS: 30
[ WARN] [1649082402.452911295]: Given stream configuration is not supported by the device!  Stream: Infrared, Stream Index: 2, Format: Y8, Width: 640, Height: 480, FPS: 30
[ INFO] [1649082402.459361845]: color stream is enabled - width: 640, height: 480, fps: 30
 04/04 15:26:42,467 WARNING [139780935542528] (backend-v4l2.cpp:809) Empty frame has arrived.
 04/04 15:26:42,467 WARNING [139780935542528] (backend-v4l2.cpp:809) Empty frame has arrived.
 04/04 15:26:42,467 WARNING [139780935542528] (backend-v4l2.cpp:809) Empty frame has arrived.
 04/04 15:26:42,467 WARNING [139780935542528] (backend-v4l2.cpp:809) Empty frame has arrived.
[ INFO] [1649082402.470200257]: publishStaticTransforms...
[ INFO] [1649082402.470303113]: RealSense Node Is Up!
 04/04 15:26:42,472 WARNING [139780838086400] (backend-v4l2.cpp:809) Empty frame has arrived.
 04/04 15:26:42,472 WARNING [139780838086400] (backend-v4l2.cpp:809) Empty frame has arrived.
 04/04 15:26:42,472 WARNING [139780838086400] (backend-v4l2.cpp:809) Empty frame has arrived.
 04/04 15:26:42,473 WARNING [139780838086400] (backend-v4l2.cpp:809) Empty frame has arrived.
 04/04 15:26:42,634 WARNING [139780935542528] (ds5-timestamp.cpp:64) UVC metadata payloads not available. Please refer to installation chapter for details.
[ WARN] [1649082402.641726662]: Frame metadata isn't available! (frame_timestamp_domain = RS2_TIMESTAMP_DOMAIN_SYSTEM_TIME)
 04/04 15:26:42,675 WARNING [139781635381120] (types.cpp:57) set(advanced_mode_preset_option) failed! Device is not in Advanced-Mode.
[ WARN] [1649082402.676006652]: Reconfigure callback failed with exception set(advanced_mode_preset_option) failed! Device is not in Advanced-Mode.:
 04/04 15:26:42,709 WARNING [139780838086400] (ds5-timestamp.cpp:64) UVC metadata payloads not available. Please refer to installation chapter for details.

[MartyG-RealSense]

The age of the configuration in this project is one where I would strongly recommend upgrading to a more recent configuration if possible. The librealsense SDK, RealSense ROS wrapper and camera firmware driver all date back to January 2018, at the start of the launch of the RealSense 400 Series cameras. Since then, a vast amount of bug fixes, enhancements and new features have been added to librealsense, the ROS wrapper and firmware driver in the subsequent years.

The age of the configuration makes it difficult to diagnose, but I would start by asking whether you have installed support for an RGBD type of launch using the instructions at #2092 (comment) (selecting the instruction for the particular ROS1 branch that you are using) as support for an RGBD launch with rs_rgbd.launch is not bundled with the RealSense ROS wrapper by default.

[RodrigoFBernardo]

This is my rs_rgbd.launch

<!--
Copyright (c) 2018 Intel Corporation

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

     http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

<!--
A launch file, derived from rgbd_launch and customized for Realsense ROS driver,
to publish XYZRGB point cloud like an OpenNI camera.

To launch Realsense with software registeration (ROS Image Pipeline and rgbd_launch):
    $ roslaunch realsense_ros_camera rs_rgbd.launch
Processing enabled by ROS driver:
    # depth rectification
Processing enabled by this node:
    # rgb rectification
    # depth registeration
    # pointcloud_xyzrgb generation

To launch Realsense with hardware registeration (ROS Realsense depth alignment):
    $ roslaunch realsense_ros_camera rs_rgbd.launch align_depth:=true
Processing enabled by ROS driver:
    # depth rectification
    # depth registration
Processing enabled by this node:
    # rgb rectification
    # pointcloud_xyzrgb generation
-->

<launch>
  <arg name="namespace"           default="camera"/>
  <arg name="manager"             default="realsense_ros_camera_manager"/>

  <!-- Camera device specific arguments -->

  <arg name="serial_no"           default=""/>
  <arg name="json_file_path"      default=""/>

  <arg name="fisheye_width"       default="640"/>
  <arg name="fisheye_height"      default="480"/>
  <arg name="enable_fisheye"      default="true"/>

  <arg name="depth_width"         default="640"/>
  <arg name="depth_height"        default="480"/>
  <arg name="enable_depth"        default="true"/>

  <arg name="infra1_width"        default="640"/>
  <arg name="infra1_height"       default="480"/>
  <arg name="enable_infra1"       default="true"/>

  <arg name="infra2_width"        default="640"/>
  <arg name="infra2_height"       default="480"/>
  <arg name="enable_infra2"       default="true"/>

  <arg name="color_width"         default="640"/>
  <arg name="color_height"        default="480"/>
  <arg name="enable_color"        default="true"/>

  <arg name="fisheye_fps"         default="30"/>
  <arg name="depth_fps"           default="30"/>
  <arg name="infra1_fps"          default="30"/>
  <arg name="infra2_fps"          default="30"/>
  <arg name="color_fps"           default="30"/>
  <arg name="gyro_fps"            default="1000"/>
  <arg name="accel_fps"           default="1000"/>
  <arg name="enable_imu"          default="true"/>

  <arg name="enable_pointcloud"   default="false"/>
  <arg name="enable_sync"         default="true"/>
  <arg name="align_depth"         default="true"/>

  <!-- rgbd_launch specific arguments -->

  <!-- Arguments for remapping all device namespaces -->
  <arg name="rgb"                             default="color" />
  <arg name="ir"                              default="infra1" />
  <arg name="depth"                           default="depth" />
  <arg name="depth_registered_pub"            default="depth_registered" />
  <arg name="depth_registered"                default="depth_registered" unless="$(arg align_depth)" />
  <arg name="depth_registered"                default="aligned_depth_to_color" if="$(arg align_depth)" />
  <arg name="depth_registered_filtered"       default="$(arg depth_registered)" />
  <arg name="projector"                       default="projector" />

  <!-- Disable bond topics by default -->
  <arg name="bond"                            default="false" />
  <arg name="respawn"                         default="$(arg bond)" />

  <!-- Processing Modules -->
  <arg name="rgb_processing"                  default="true"/>
  <arg name="debayer_processing"              default="false" />
  <arg name="ir_processing"                   default="false"/>
  <arg name="depth_processing"                default="false"/>
  <arg name="depth_registered_processing"     default="true"/>
  <arg name="disparity_processing"            default="false"/>
  <arg name="disparity_registered_processing" default="false"/>
  <arg name="hw_registered_processing"        default="$(arg align_depth)" />
  <arg name="sw_registered_processing"        default="true" unless="$(arg align_depth)" />
  <arg name="sw_registered_processing"        default="false" if="$(arg align_depth)" />

  <group ns="$(arg namespace)">

    <!-- Launch the camera device nodelet-->
    <include file="$(find realsense_ros_camera)/launch/includes/nodelet.launch.xml">
      <arg name="manager"                  value="$(arg manager)"/>
      <arg name="serial_no"                value="$(arg serial_no)"/>
      <arg name="json_file_path"           value="$(arg json_file_path)"/>

      <arg name="enable_pointcloud"        value="$(arg enable_pointcloud)"/>
      <arg name="enable_sync"              value="$(arg enable_sync)"/>
      <arg name="align_depth"              value="$(arg align_depth)"/>

      <arg name="fisheye_width"            value="$(arg fisheye_width)"/>
      <arg name="fisheye_height"           value="$(arg fisheye_height)"/>
      <arg name="enable_fisheye"           value="$(arg enable_fisheye)"/>

      <arg name="depth_width"              value="$(arg depth_width)"/>
      <arg name="depth_height"             value="$(arg depth_height)"/>
      <arg name="enable_depth"             value="$(arg enable_depth)"/>

      <arg name="color_width"              value="$(arg color_width)"/>
      <arg name="color_height"             value="$(arg color_height)"/>
      <arg name="enable_color"             value="$(arg enable_color)"/>

      <arg name="infra1_width"             value="$(arg infra1_width)"/>
      <arg name="infra1_height"            value="$(arg infra1_height)"/>
      <arg name="enable_infra1"            value="$(arg enable_infra1)"/>

      <arg name="infra2_width"             value="$(arg infra2_width)"/>
      <arg name="infra2_height"            value="$(arg infra2_height)"/>
      <arg name="enable_infra2"            value="$(arg enable_infra2)"/>

      <arg name="fisheye_fps"              value="$(arg fisheye_fps)"/>
      <arg name="depth_fps"                value="$(arg depth_fps)"/>
      <arg name="infra1_fps"               value="$(arg infra1_fps)"/>
      <arg name="infra2_fps"               value="$(arg infra2_fps)"/>
      <arg name="color_fps"                value="$(arg color_fps)"/>
      <arg name="gyro_fps"                 value="$(arg gyro_fps)"/>
      <arg name="accel_fps"                value="$(arg accel_fps)"/>
      <arg name="enable_imu"               value="$(arg enable_imu)"/>
    </include>

    <!-- RGB processing -->
    <include if="$(arg rgb_processing)"
             file="$(find rgbd_launch)/launch/includes/rgb.launch.xml">
      <arg name="manager"                       value="$(arg manager)" />
      <arg name="respawn"                       value="$(arg respawn)" />
      <arg name="rgb"                           value="$(arg rgb)" />
      <arg name="debayer_processing"            value="$(arg debayer_processing)" />
    </include>

    <group if="$(eval depth_registered_processing and sw_registered_processing)">
      <node pkg="nodelet" type="nodelet" name="register_depth"
            args="load depth_image_proc/register $(arg manager) $(arg bond)" respawn="$(arg respawn)">
        <remap from="rgb/camera_info"             to="$(arg rgb)/camera_info" />
        <remap from="depth/camera_info"           to="$(arg depth)/camera_info" />
        <remap from="depth/image_rect"            to="$(arg depth)/image_rect_raw" />
        <remap from="depth_registered/image_rect" to="$(arg depth_registered)/sw_registered/image_rect_raw" />
      </node>

      <!-- Publish registered XYZRGB point cloud with software registered input -->
      <node pkg="nodelet" type="nodelet" name="points_xyzrgb_sw_registered"
            args="load depth_image_proc/point_cloud_xyzrgb $(arg manager) $(arg bond)" respawn="$(arg respawn)">
        <remap from="rgb/image_rect_color"        to="$(arg rgb)/image_rect_color" />
        <remap from="rgb/camera_info"             to="$(arg rgb)/camera_info" />
        <remap from="depth_registered/image_rect" to="$(arg depth_registered_filtered)/sw_registered/image_rect_raw" />
        <remap from="depth_registered/points"     to="$(arg depth_registered)/points" />
      </node>
    </group>

    <group if="$(eval depth_registered_processing and hw_registered_processing)">
      <!-- Publish registered XYZRGB point cloud with hardware registered input (ROS Realsense depth alignment) -->
      <node pkg="nodelet" type="nodelet" name="points_xyzrgb_hw_registered"
            args="load depth_image_proc/point_cloud_xyzrgb $(arg manager) $(arg bond)" respawn="$(arg respawn)">
        <remap from="rgb/image_rect_color"        to="$(arg rgb)/image_rect_color" />
        <remap from="rgb/camera_info"             to="$(arg rgb)/camera_info" />
        <remap from="depth_registered/image_rect" to="$(arg depth_registered)/image_raw" />
        <remap from="depth_registered/points"     to="$(arg depth_registered_pub)/points" />
      </node>
    </group>

  </group>

</launch>

Rodrigo Bernardo

[RodrigoFBernardo]

The framerate of of the topics I am using:

$ rostopic hz /camera/color/image_raw

subscribed to [/camera/color/image_raw]
average rate: 12.763
	min: 0.078s max: 0.079s std dev: 0.00036s window: 11

$ rostopic hz /camera/color/camera_info
subscribed to [/camera/color/camera_info]
average rate: 29.852
	min: 0.031s max: 0.035s std dev: 0.00133s window: 28

[MartyG-RealSense]

Seeing that kind of difference between the color and depth FPS, I would recommend trying to disable an RGB option called auto_exposure_priority (a separate option from auto_exposure) to see whether it makes a positive difference. If auto_exposure is enabled and auto_exposure_priority is disabled then it forces the streams to try to maintain a constant FPS rate instead of being allowed to vary.

You could test quickly whether the problem is related to auto_exposure_priority by opening the dynamic reconfigure interface window after launch, going to the rgb_camera category of options and then finding and disabling the auto_exposure_priority setting.

The section of the ROS wrapper documentation at the link below explains how to open the dynamic reconfigure interface after launch has completed by using the instruction rosrun rqt_reconfigure rqt_reconfigure

https://github.com/IntelRealSense/realsense-ros#set-camera-controls-using-dynamic-reconfigure-params

[RodrigoFBernardo]

I am using two machines, on my master I have no difference between the topics, I only have on my slave. This difference might be associated with the internet connection, which is over Wifi. I will change this parameter to see if it improves.

[MartyG-RealSense]

Hi @RodrigoCodeBernardo Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.