SegmentationMapping

• Label pointcloud from stereo cameras or lidar with Tensorflow trained neural nets graph.pb
• Build 3D Semantic voxel maps, 2D occupancy maps, 2D obstacle distance maps

The Youtube video of the Cassie Bipedal robot's perception and mapping system is powered by this library.

Here is a semantic map built on NCLT dataset

Pipeline

Two types of accepted inputs (from ROS topics):

• Raw RGBD images from depth cameras, or
• Lidar scans + RGB images.

C++: pcl, eigen, xmlrpcpp

Outputs (in ROS messages):

• Labeled images, each pixel with a distribution across classes
• Labeled PointCloud ROS message
• Semantic Octomap
• 2D occupancy maps
• 2D cost map, cost computed from distance to closest obstacles

Performance

Tested on

• AlienWare Laptop (running in realtime for the Cassie Blue experiement): 3FPS on i7-6820, 32GB RAM, GTX1070
• Dell XPS: 5FPS on i7-7700, 32GB RAM, GTX1070.
• Jetson Xavier: Around 4Hz

Dependencies

ROS: Tested on Kinetic + Ubuntu 16.04, Melodic + Ubuntu 18.04

Neural Network:

• tensorflow-gpu-1.8.0
• tensorrt4
• cuda-9.0,
• cudnn-7

Python thirdparty:

• python-pcl
• python-opencv

C++: pcl, eigen, OpenCV

ros thirdparty:

• semantic octomap: Modified octomap, supporting Bayesian updates for semantic label fusion
• octomap_ros: Put it in the same catkin workspace as the SegmentationMappping repo.
• octomap_msgs: Put it in the same catkin workspace as the SegmentationMappping repo.
• ros_numpy: Put it in the same catkin workspace as the SegmentationMappping repo.

Compiling

Make sure that your semantic octomap is build. Make sure that you haven't install the native octomap and octomap_ros from apt-get. Then run catkin_make install -DCMAKE_BUILD_TYPE=Release -Doctomap_DIR=${you_octomap_source_directory}/lib/cmake/octomap/ --pkg SegmentationMapping

launchfiles

• run on Cassie with a single stereo camera (Intel Realsense): roslaunch SegmentationMapping cassie_stereo_py.launch
• run nclt dataset: roslaunch SegmentationMapping nclt_distribution_deeplab.launch
• run on mulitple cameras with multiple segmentations at the same time: roslaunch SegmentationMapping cassie_stereo_py_multi_camera.launch