Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New version of realsense for pointcloud it is hanging rviz #1886

Closed
ranjitkathiriya opened this issue May 20, 2021 · 36 comments
Closed

New version of realsense for pointcloud it is hanging rviz #1886

ranjitkathiriya opened this issue May 20, 2021 · 36 comments
Labels

Comments

@ranjitkathiriya
Copy link

Hello there,

I have just now installed the latest version of realsense_ros and I have tried various version which was greater than 2.2.22. When I visualize in rviz I see my rviz is hanging and fps is getting around 2 or 0.

Can you help me to resolve this issue? What may be the issue over here? It was working perfectly before with the older version.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented May 20, 2021

Hi @ranjitkathiriya Each RealSense ROS wrapper version should be matched with a specific librealsense SDK version recommended in the wrapper release notes for that version. For example, RealSense ROS wrapper 2.2.24 and librealsense SDK 2.44.0.

https://github.com/IntelRealSense/realsense-ros/releases

When updating the wrapper version, have you first installed the recommended librealsense version, and then secondy built the ROS wrapper once librealsense has been successfully installed?

Every time that librealsense is updated to a new version, the ROS wrapper should be built again afterwards.

@ranjitkathiriya
Copy link
Author

How should I completely remove LibRealSense v2.36.0?

setting /run_id to d88a3c46-b987-11eb-9577-33c0391d24de
process[rosout-1]: started with pid [221577]
started core service [/rosout]
process[camera/realsense2_camera_manager-2]: started with pid [221584]
process[camera/realsense2_camera-3]: started with pid [221585]
process[camera/color_rectify_color-4]: started with pid [221586]
process[camera/points_xyzrgb_hw_registered-5]: started with pid [221587]
[ INFO] [1621527867.963164850]: Initializing nodelet with 16 worker threads.
[ INFO] [1621527868.140611453]: RealSense ROS v2.3.0
[ INFO] [1621527868.140683288]: Built with LibRealSense v2.36.0
[ INFO] [1621527868.140713826]: Running with LibRealSense v2.45.0
[ WARN] [1621527868.140736308]: ***************************************************
[ WARN] [1621527868.140744404]: ** running with a different librealsense version **
[ WARN] [1621527868.140752369]: ** than the one the wrapper was compiled with!   **
[ WARN] [1621527868.140760594]: ***************************************************
[ INFO] [1621527868.173551755]:  
[ INFO] [1621527868.180043051]: Device with serial number 941322071646 was found.

[ INFO] [1621527868.180140965]: Device with physical ID /sys/devices/pci0000:00/0000:00:08.1/0000:05:00.3/usb4/4-2/4-2:1.0/video4linux/video0 was found.
[ INFO] [1621527868.180161133]: Device with name Intel RealSense D435I was found.
[ INFO] [1621527868.182245462]: Device with port number 4-2 was found.
[ INFO] [1621527868.182311737]: Device USB type: 3.2

@MartyG-RealSense
Copy link
Collaborator

What this message is basically saying is that now that you have installed librealsense 2.45.0, build ROS wrapper 2.3.0 again and then this message should disappear.

This is because if you build the wrapper with a particular librealsense version and then update librealsense, the wrapper has to be built again afterwards or this "built with, running with" message occurs.

@ranjitkathiriya
Copy link
Author

Yes I have done build things with 2.3.0

catkin build
source devel/setup.bash

Then also I am facing the same issue..

@MartyG-RealSense
Copy link
Collaborator

You mean that it is still stating Built with LibRealSense v2.36.0, Running with LibRealSense v2.45.0 even though you have rebuilt the wrapper?

@ranjitkathiriya
Copy link
Author

Yes,

[ INFO] [1621529346.473334607]: Initializing nodelet with 16 worker threads.
[ INFO] [1621529346.621926517]: RealSense ROS v2.3.0
[ INFO] [1621529346.621991400]: Built with LibRealSense v2.36.0
[ INFO] [1621529346.622024295]: Running with LibRealSense v2.45.0
[ WARN] [1621529346.622059635]: ***************************************************
[ WARN] [1621529346.622085686]: ** running with a different librealsense version **
[ WARN] [1621529346.622111737]: ** than the one the wrapper was compiled with!   **
[ WARN] [1621529346.622137998]: ***************************************************
[ INFO] [1621529346.654397024]:

@MartyG-RealSense
Copy link
Collaborator

Thank you. And was librealsense installed with Debian packages or built from source code?

@ranjitkathiriya
Copy link
Author

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented May 20, 2021

You have installed from Debian packages then. A RealSense team member provides advice in the link below for uninstalling librealsense packages.

IntelRealSense/librealsense#7052 (comment)

sudo apt purge librealsense2 -y
sudo apt update
sudo apt install librealsense2-utils librealsense2-*-dev librealsense2-*-dbg

@jbeck28
Copy link

jbeck28 commented May 20, 2021

I have run into the same issue after updating, and have followed the instructions in your last comment.

[ INFO] [1621536476.241420284]: RealSense ROS v2.3.0 [ INFO] [1621536476.241469303]: Built with LibRealSense v2.45.0 [ INFO] [1621536476.241497407]: Running with LibRealSense v2.45.0 [ INFO] [1621536476.280627182]:

@MartyG-RealSense
Copy link
Collaborator

Hi @jbeck28 Your version details look correct, as both Built With and Running With have the same librealsense version number. Your RealSense ROS version of 2.3.0 is also the correct one for use with 2.45.0.

@jbeck28
Copy link

jbeck28 commented May 20, 2021

That is promising. I however, still have the issue of slow frame-rates, and when using a simple chessboardfinder, there are NaN's now where previously there were not.

Any other ideas on what may resolve these problems?

@MartyG-RealSense
Copy link
Collaborator

@jbeck28 In case investigation of your problems turns out to be complex, I recommend creating a new issue by visiting the link below and clicking the green New Issue button. Please include in the message which camera model you are using. Thanks!

https://github.com/IntelRealSense/realsense-ros/issues

@ranjitkathiriya
Copy link
Author

Hello @MartyG-RealSense ,

I have followed your steps then also I am getting this warning. Is there any way then I can manually delete the file or I can change the build path?

[ INFO] [1621583636.718093947]: Initializing nodelet with 16 worker threads.
[ INFO] [1621583637.126554822]: RealSense ROS v2.3.0
[ INFO] [1621583637.126615374]: Built with LibRealSense v2.36.0
[ INFO] [1621583637.126645406]: Running with LibRealSense v2.45.0
[ WARN] [1621583637.126677602]: ***************************************************
[ WARN] [1621583637.126702745]: ** running with a different librealsense version **
[ WARN] [1621583637.126724256]: ** than the one the wrapper was compiled with!   **
[ WARN] [1621583637.126748071]: ***************************************************
[ INFO] [1621583637.154385896]:  
[ INFO] [1621583637.159516249]: Device with serial number 026322072776 was found.

@MartyG-RealSense
Copy link
Collaborator

Hi @ranjitkathiriya Have you tried the purge and install method that I suggested yesterday, please?

#1886 (comment)

@ranjitkathiriya
Copy link
Author

YesI tried this all command and tried

sudo apt purge librealsense2 -y
sudo apt update
sudo apt install librealsense2-utils librealsense2-*-dev librealsense2-*-dbg

@ranjitkathiriya
Copy link
Author

ranjitkathiriya commented May 21, 2021

Yes, Solved by removing all build, devel, logs folder and building catkin_ws again. Now it is solved. Other things I am getting this warning it is normal?

auto-starting new master
process[master]: started with pid [61603]
ROS_MASTER_URI=http://localhost:11311

setting /run_id to d2a01614-ba0b-11eb-b341-153f9908e0db
process[rosout-1]: started with pid [61628]
started core service [/rosout]
process[camera/realsense2_camera_manager-2]: started with pid [61635]
process[camera/realsense2_camera-3]: started with pid [61636]
process[camera/color_rectify_color-4]: started with pid [61637]
process[camera/points_xyzrgb_hw_registered-5]: started with pid [61638]
[ INFO] [1621584551.645386166]: Initializing nodelet with 16 worker threads.
[ INFO] [1621584551.987665033]: RealSense ROS v2.3.0
[ INFO] [1621584551.987701910]: Built with LibRealSense v2.45.0
[ INFO] [1621584551.987715600]: Running with LibRealSense v2.45.0
[ INFO] [1621584552.015852437]:  
[ INFO] [1621584552.020228251]: Device with serial number 026322072776 was found.

[ INFO] [1621584552.020588577]: Device with physical ID /sys/devices/pci0000:00/0000:00:08.1/0000:05:00.3/usb4/4-1/4-1:1.0/video4linux/video0 was found.
[ INFO] [1621584552.020918731]: Device with name Intel RealSense D435I was found.
[ INFO] [1621584552.021628209]: Device with port number 4-1 was found.
[ INFO] [1621584552.022029116]: Device USB type: 3.2
[ INFO] [1621584552.023840737]: getParameters...
[ INFO] [1621584552.102351502]: setupDevice...
[ INFO] [1621584552.102396970]: JSON file is not provided
[ INFO] [1621584552.102437340]: ROS Node Namespace: camera
[ INFO] [1621584552.102461157]: Device Name: Intel RealSense D435I
[ INFO] [1621584552.102485882]: Device Serial No: 026322072776
[ INFO] [1621584552.102507813]: Device physical port: /sys/devices/pci0000:00/0000:00:08.1/0000:05:00.3/usb4/4-1/4-1:1.0/video4linux/video0
[ INFO] [1621584552.102533586]: Device FW version: 05.12.13.50
[ INFO] [1621584552.102562711]: Device Product ID: 0x0B3A
[ INFO] [1621584552.102589042]: Enable PointCloud: On
[ INFO] [1621584552.102623336]: Align Depth: On
[ INFO] [1621584552.102656232]: Sync Mode: On
[ INFO] [1621584552.102726915]: Device Sensors: 
[ INFO] [1621584552.106294700]: Stereo Module was found.
 21/05 09:09:12,107 WARNING [140215151855360] (backend-v4l2.cpp:1357) Pixel format 36315752-1a66-a242-9065-d01814a likely requires patch for fourcc code RW16!
[ INFO] [1621584552.115945712]: RGB Camera was found.
[ INFO] [1621584552.116197850]: Motion Module was found.
[ INFO] [1621584552.116242132]: (Confidence, 0) sensor isn't supported by current device! -- Skipping...
[ INFO] [1621584552.116685433]: Add Filter: colorizer
[ INFO] [1621584552.118713990]: Add Filter: pointcloud
[ INFO] [1621584552.119105747]: num_filters: 3
[ INFO] [1621584552.119137177]: Setting Dynamic reconfig parameters.
[ WARN] [1621584552.153916554]: Param '/camera/stereo_module/auto_exposure_limit' has value 0 that is not in range [1, 165000]. Removing this parameter from dynamic reconfigure options.
[ WARN] [1621584552.154957373]: Param '/camera/stereo_module/auto_gain_limit' has value 0 that is not in range [16, 248]. Removing this parameter from dynamic reconfigure options.
[ WARN] [1621584552.166880420]: Param '/camera/rgb_camera/power_line_frequency' has value 3 that is not in the enum { {50Hz: 1} {60Hz: 2} {Disabled: 0} }. Removing this parameter from dynamic reconfigure options.
[ INFO] [1621584552.189416515]: Done Setting Dynamic reconfig parameters.
[ INFO] [1621584552.190846436]: depth stream is enabled - width: 848, height: 480, fps: 30, Format: Z16
[ INFO] [1621584552.192148264]: color stream is enabled - width: 1280, height: 720, fps: 30, Format: RGB8
[ INFO] [1621584552.194334669]: setupPublishers...
[ INFO] [1621584552.198396882]: Expected frequency for depth = 30.00000
[ INFO] [1621584552.213746559]: Expected frequency for color = 30.00000
[ INFO] [1621584552.236792377]: Expected frequency for aligned_depth_to_color = 30.00000
[ INFO] [1621584552.250160153]: setupStreams...
[ INFO] [1621584552.250740628]: insert Depth to Stereo Module
[ INFO] [1621584552.250873682]: insert Color to RGB Camera
[ INFO] [1621584552.307274752]: SELECTED BASE:Depth, 0
[ INFO] [1621584552.317769623]: RealSense Node Is Up!
[ WARN] [1621584552.459166150]: 
[ERROR] [1621584593.374277042]: Depth image has unsupported encoding [rgb8]
[ INFO] [1621584600.926207113]: publishing unordered pointcloud.
[ WARN] [1621584608.134205459]: No stream match for pointcloud chosen texture Process - Color
[ WARN] [1621584608.341832384]: No stream match for pointcloud chosen texture Process - Color
[ WARN] [1621584615.316117746]: No stream match for pointcloud chosen texture Process - Color
[ WARN] [1621584615.483069250]: No stream match for pointcloud chosen texture Process - Color

What should be the minimum FPS in rviz again I am getting around 4 or 5.

aaaaa

And In real sense viewer, this is my frames drop per second graphs.

Screenshot from 2021-05-21 10-11-24

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented May 21, 2021

I would not recommend enabling RGB, Depth and Motion Module all at the same time in the RealSense Viewer. If you use 3 streams, when the Motion Module is enabled then it can cause the RGB stream to become No Frames Received. It does not matter which order the streams are enabled in, because if the problem occurs then the trigger for it is always the enabling of the Motion Module. So I would suggest only enabling RGB and Depth at the same time if possible.

There is no fix that can correct the problem if it happens to you except for rebooting the computer.

Your FPS problem in RViz sounds like the one jbeck28 was experiencing with rs_rgbd.launch.

#1887

@ranjitkathiriya
Copy link
Author

ranjitkathiriya commented May 21, 2021

I am experiencing this command.

roslaunch realsense2_camera rs_rgbd.launch filters:=colorizer,pointcloud align_depth:=true

I have restarted my pc I have a clear catch and everything I have tried. Then also I am not able to fix this issue. In RVIZ I am getting fps in between 2-8.

@MartyG-RealSense
Copy link
Collaborator

What kind of point cloud do you get if you remove align_depth:=true from the roslaunch instruction?

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented May 21, 2021

if an organized cloud has invalid points and so is_dense = false then do you need to include is_dense:=false in the roslaunch instruction for roslaunch realsense2_camera rs_rgbd.launch filters:=colorizer,pointcloud ordered_pc:=true - Organized is_dense=False ?

Also, if you are now able to set ordered_pc:=true from wrapper version 2.2.22 onward then could you instead use roslaunch realsense2_camera rs_camera and set the default unordered cloud in the rs_camera mode to ordered, if the need for an ordered cloud is the main reason for you using rs_rgbd

@ranjitkathiriya
Copy link
Author

I am sorry for the mistake I have updated the question again.
#1886 (comment)

@ranjitkathiriya
Copy link
Author

roslaunch realsense2_camera rs_camera.launch filters:=colorizer,pointcloud ordered_pc:=true align_depth:=true

  1. If I am not maintaining align_depth:=true then I am not able to view the colour depth stream.
  2. when I run rostopic echo /camera/depth/color/points
    Screenshot from 2021-05-21 11-37-36

And When I run this command:

roslaunch realsense2_camera rs_camera.launch filters:=colorizer,pointcloud align_depth:=true

Screenshot from 2021-05-21 11-39-03

What I wanted for my project:
I need Pointcloud with good FPS in RVIZ, COLOR depth channel and RGB images for my project. For getting an aligned colour depth channel I think I have to use this align_depth:=true.

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented May 21, 2021

The pointcloud example in the RealSense ROS documentation seems to map RGB color onto depth on a pointcloud pretty well whilst only using roslaunch realsense2_camera rs_camera.launch filters:=pointcloud

https://github.com/IntelRealSense/realsense-ros#point-cloud

image

@ranjitkathiriya
Copy link
Author

ranjitkathiriya commented May 21, 2021

The pointcloud example in the RealSense ROS documentation seems to map RGB color onto depth on a point cloud pretty well whilst only using roslaunch realsense2_camera rs_camera.launch filters:=pointcloud

Yes, This works perfectly for me even I am getting 28-31 fps with this command.

roslaunch realsense2_camera rs_camera.launch filters:=pointcloud,colorizer align_depth:=true I want to run this command because I am doing two-way processing 2 sides.

  1. On the RGB camera stream I am doing Object detection and on the coloured depth stream I am identifying the object based on shape and pose into the point cloud. This coordination that was identified by colour depth stream I am giving quaternion to my robot and my robot is navigating based on that location.
  2. When I am running align_depth:=true and filters:=pointcloud,colorizer at that time I am able to get depth colour stream.
  3. When I am running this command at that time I am able to see point cloud processing is a bit slow and due to that, I am able to get FPS around 4-8.

Over here I am calculating the pose, angle, and centre point for a particular object from coloured depth stream.
Note: this is the second phase of my algorithm before that My object detection was identifying the object and this custom is working completely on colourized depth stream.

R_1
R_1

You can see the green point in the second Image. It is calculated with the help of colourized depth stream.

My question was: is there any way that I can get point cloud much faster with rgb depth stream?

@MartyG-RealSense
Copy link
Collaborator

The warning No stream match for pointcloud chosen texture Process - Color that you are experiencing was discussed recently in the link below.

#1826

Doronhi said in that discussion that as far as he knew the issue emerged in librealsense 2.43.0 and seemed to be based on an issue in librealsense that was not currently fixed.

#1826 (comment)

@doronhi
Copy link
Contributor

doronhi commented May 23, 2021

Hi,
My comment relates to a pointcloud from a recorded rosbag file. This issue refers to live-streaming so I don't think it is relevant.
Using align_depth, colorizer, and pointcloud all together requires more resources, naturally.
A way to reduce computation might be to reduce the resolution.
In order to make things clearer, and help us look for ways to improve performance, could you post a list of the topics you subscribe to?

@ranjitkathiriya
Copy link
Author

ranjitkathiriya commented May 24, 2021

In order to make things clearer, and help us look for ways to improve performance, could you post a list of the topics you subscribe to?

Thanks for the reply, @doronhi.

Yes, sure.

I required this following topics,
/camera/color/image_raw (RGB - for object detection )
/camera/aligned_depth_to_color/image_raw (RGB DEPTH - for identifying features from the detected object as describe above.)

/camera/depth/camera_info - All camera information.
/camera/depth/color/points - Point cloud (organized, is_dense=False) for converting 2d object to 3d and to navigate my robot into 3d space with respect to pointcloud.

Screenshot from 2021-05-24 09-08-07

I am getting low FPS, Can you please help me with how to reduce resolution and what is a suitable resolution for subscribing to all this above topic with appropriate FPS and speed.

Thanks!

@doronhi
Copy link
Contributor

doronhi commented May 24, 2021

The list of topics you added let me think that an improvement in performance can be made - the colorizer filter is currently applied to both depth and aligned-depth-to-color images, although you use only one of them. This, however, requires some coding, possibly messy and hacky looking...

The README.md file describes the parameters used for setting stream resolutions. For example
you can try the following command: roslaunch realsense2_camera rs_camera.launch depth_width:=640 depth_height:=480 depth_fps:=30 and add the rest of your filters etc.
For a full list of available stream formats type rs-enumerate-devices, if you have librealsense2 installed, or consult with the datasheet

The FPS is really machine-dependent. On my machine, I get a little more the 28fps with your configuration so you'll have to try and see if you can find a balance between fps and resolution that is good enough for you.
Also, I think that using is_dense=True for your pointcloud may also reduce some load.

@ranjitkathiriya
Copy link
Author

ranjitkathiriya commented May 24, 2021

The list of topics you added let me think that an improvement in performance can be made - the colorizer filter is currently applied to both depth and aligned-depth-to-color images, although you use only one of them.

Thanks! for replying and yes, I am using aligned depth only.

The FPS is really machine-dependent. On my machine, I get a little more the 28fps with your configuration so you'll have to try and see if you can find a balance between fps and resolution that is good enough for you.

28 FPS is good! Well, I am using the below line to obtain topics.

roslaunch realsense2_camera rs_camera.launch filters:=pointcloud,colorizer align_depth:=true ordered_pc:=true

is_dense=True

I am obtaining distance from an object through a pinhole camera and converting 2d points to 3d point in ROS. For that, I need an organized PC that. I also have tried to obtain distance with depth channel but it was not that good and accurate as compare to PC. https://answers.ros.org/question/372663/convert-pixel-coordinates-uv-to-pointcloud2-x-y-z-python/#372672

Even with a normal aligned stream, I can see RVIZ is hanging. I think it's an issue with the align_depth flag whenever I am setting it to true then it decreases FPS. roslaunch realsense2_camera rs_rgbd.launch align_depth:=true

I have a good system: https://rog.asus.com/laptops/rog-zephyrus/rog-zephyrus-g15-series/

@rowanborder
Copy link

rowanborder commented May 24, 2021

Hi, I am having this exact same issue after upgrading to realsense-ros 2.3.0 from 2.2.20 . In the previous version I was able to use the pointcloud filter and align_depth:=true and received ~30fps in RViz when visualising the pointcloud. Since updating to 2.3.0, using exactly the same configuration file, the performance in RViz has dropped to ~5fps.

I have traced the breaking change to this commit, which appears to change which depth image the pointcloud is obtained from.

@doronhi
Copy link
Contributor

doronhi commented May 25, 2021

Since version 2.2.24, setting align_depth:=true also creates the pointcloud aligned to the color image. That means, it will have a point for every pixel in the color image, instead of the default behavior of having a point for every pixel in the depth image.
Assuming you did not set color and depth resolutions, you now have a pointcloud for 1280x720 pixels (default color frame in D435) instead of 848x480 pixels - the default for depth image.
Running the following command and the fps in RViz comes back to 30, on my machine.

roslaunch realsense2_camera rs_camera.launch align_depth:=true color_width:=848 color_height:=480 color_fps:=30 filters:=pointcloud

The said modification did not assume you would need a pointcloud aligned to the depth image and aligned_depth_to_color image instead of the original depth image. Is this the case for you?

@rowanborder
Copy link

I'm actually using the L515. I reduced the color resolution to 960x540 instead of 1280x720 and reduced the depth resolution from 1024x768 to 640x480 and that is visualising at 30fps in RViz.

For my application I don't need the pointcloud and depth image in the same frame, just the depth and color images in a common frame. As the color image is almost always higher resolution than the depth image, how are depth values for the extra color pixels calculated?

@ranjitkathiriya
Copy link
Author

ranjitkathiriya commented May 25, 2021

My issue is solved and I am getting 20-25 FPS. When I am subscribing topic in RVIZ at that time it is making my RVIZ slow. What I am doing I am just processing it internally through code and Just focusing on output. For test only I am using rviz.

@ranjitkathiriya
Copy link
Author

Can I close this case?

@MartyG-RealSense
Copy link
Collaborator

As you are the creator of the case @ranjitkathiriya and there are no other matters linked to this case, it is fine to close it if you are satisifed with the outcome. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants