BKISemanticMapping

Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping

Quantitative results on SemanticKITTI dataset sequence 00-10 for 19 semantic classes. SqueezeSegV2-kNN (Sq.-kNN)

Sequence	Method	Car	Bicycle	Motorcycle	Truck	Other Vehicle	Person	Bicyclist	Motorcyclist	Road	Parking	Sidewalk	Other Ground	Building	Fence	Vegetation	Trunk	Terrain	Pole	Traffic Sign	Average
00	Sq.-KNN	92.1	18.3	55.0	76.5	62.9	34.2	52.0	61.4	94.7	71.0	87.9	1.2	89.8	54.6	82.2	53.1	79.3	38.6	51.5	60.9
	S-CMS	95.6	23.5	69.8	88.3	74.4	47.9	71.6	56.9	96.3	78.1	91.2	3.1	93.6	64.2	87.4	70.1	83.5	61.1	70.7	69.9
	S-BKI	96.9	26.5	75.8	93.5	80.1	61.5	77.5	71.0	96.2	79.2	91.5	6.6	94.6	66.5	88.9	73.4	84.5	65.8	76.2	75.0
01	Sq.-KNN	83.8	n/a	n/a	n/a	82.9	n/a	n/a	67.9	92.6	n/a	n/a	70.5	58.0	71.4	72.1	18.0	71.5	21.8	68.9	64.0
	S-CMS	89.8	n/a	n/a	n/a	91.0	n/a	n/a	70.3	93.4	n/a	n/a	74.2	64.4	73.8	75.1	26.3	74.7	31.9	78.7	70.3
	S-BKI	91.0	n/a	n/a	n/a	96.0	n/a	n/a	70.7	94.3	n/a	n/a	75.2	67.1	75.1	76.4	30.6	76.1	36.2	81.4	72.5
02	Sq.-KNN	90.9	14.5	50.8	n/a	56.4	38.6	n/a	59.9	93.9	68.1	84.9	50.9	79.1	66.1	82.5	48.9	68.3	25.7	35.9	59.7
	S-CSM	95.4	28.5	73.4	n/a	80.3	60.3	n/a	75.1	94.8	74.4	87.4	61.7	85.0	71.8	86.7	66.9	72.9	43.5	55.7	71.4
	S-BKI	95.8	31.1	76.4	n/a	83.3	62.5	n/a	79.5	94.8	75.0	87.4	63.6	85.6	72.1	87.1	68.8	73.4	45.9	60.4	73.1
03	Sq.-KNN	88.4	21.9	n/a	12.4	60.1	16.3	n/a	n/a	92.8	57.9	83.2	n/a	77.4	70.1	79.3	41.6	62.3	35.9	47.3	56.5
	S-CSM	92.4	29.7	n/a	23.1	65.4	17.6	n/a	n/a	94.3	69.4	86.9	n/a	80.4	73.8	83.2	52.3	66.9	53.5	62.0	63.0
	S-BKI	94.5	42.4	n/a	48.8	73.6	23.8	n/a	n/a	94.3	73.2	87.2	n/a	82.1	74.7	84.1	55.7	67.4	57.3	66.7	68.0
04	Sq.-KNN	84.9	n/a	n/a	n/a	68.1	20.8	n/a	n/a	95.8	26.1	68.4	61.5	49.3	76.4	82.6	14.0	67.6	36.0	44.6	56.9
	S-CSM	88.3	n/a	n/a	n/a	71.2	23.2	n/a	n/a	96.5	40.5	72.5	64.0	52.1	78.5	85.5	19.4	72.5	50.8	57.6	62.3
	S-BKI	87.7	n/a	n/a	n/a	82.5	37.3	n/a	n/a	96.2	55.7	72.3	68.3	56.9	80.3	87.1	24.4	72.7	55.5	67.0	67.5
05	Sq.-KNN	89.1	8.6	15.4	82.5	70.9	31.0	55.0	n/a	94.7	84.8	85.0	61.5	87.0	72.4	75.5	30.3	64.6	27.6	39.5	59.8
	S-CSM	93.4	15.4	28.9	86.4	78.4	39.8	69.4	n/a	96.4	90.1	88.5	70.0	90.9	77.7	81.2	46.8	69.5	47.6	57.4	68.2
	S-BKI	93.2	27.4	46.0	89.0	84.1	47.5	83.3	n/a	94.2	88.0	83.6	75.2	92.4	75.3	82.1	53.5	69.5	50.2	63.3	72.1
06	Sq.KNN	85.4	17.1	50.2	86.7	66.1	27.6	64.3	n/a	87.6	56.0	74.9	66.5	83.9	38.4	61.9	32.0	89.5	40.1	52.7	60.1
	S-CSM	91.8	22.7	62.5	89.8	75.4	43.3	92.1	n/a	91.1	68.2	80.4	70.5	89.4	49.3	69.7	50.1	92.2	60.0	77.9	70.9
	S-BKI	92.6	28.7	67.9	93.5	81.4	62.7	95.4	n/a	90.3	70.7	79.9	71.8	91.7	53.6	73.7	54.7	91.9	66.4	84.8	75.1
07	Sq.-KNN	92.4	21.3	64.0	83.6	69.8	53.2	63.6	n/a	93.9	75.9	89.3	n/a	90.9	59.7	76.5	45.9	82.8	40.2	54.0	68.1
	S-CSM	94.9	25.9	76.8	82.6	81.5	64.2	88.0	n/a	95.8	80.9	92.0	n/a	93.8	66.6	80.8	59.8	84.7	55.4	73.2	76.2
	S-BKI	93.8	29.2	80.2	82.7	87.8	70.1	92.7	n/a	93.9	77.0	87.7	n/a	94.1	63.4	81.4	84.1	84.5	53.2	77.6	77.2
08	Sq.-KNN	86.7	14.4	24.6	21.0	23.3	23.5	40.9	n/a	90.1	32.4	74.8	1.2	79.6	42.7	79.2	36.5	71.1	28.3	24.8	44.1
	S-CSM	90.5	23.0	34.9	26.8	29.1	32.4	49.4	n/a	92.6	38.7	79.0	1.1	84.6	51.6	83.3	48.3	72.9	44.1	31.6	50.8
	S-BKI	92.3	30.0	39.7	29.3	32.1	38.8	54.7	n/a	92.9	40.9	79.9	1.1	86.6	54.6	84.9	52.3	74.2	47.9	34.7	53.7
09	Sq.-KNN	89.2	5.3	48.0	79.8	61.3	37.3	n/a	n/a	91.0	59.0	79.9	38.9	80.9	62.9	77.0	32.3	61.7	31.8	52.6	58.2
	S-CSM	93.9	12.2	71.9	85.6	71.6	47.5	n/a	n/a	91.8	67.0	83.1	23.4	88.9	65.7	82.6	42.9	64.9	52.4	53.0	64.6
	S-BKI	96.0	22.8	80.2	90.5	79.7	60.7	n/a	n/a	91.7	70.0	83.8	30.7	90.8	69.1	84.0	46.3	66.0	59.1	58.2	69.4
10	Sq.-KNN	84.0	8.1	36.2	49.3	10.2	40.9	n/a	n/a	89.4	59.6	78.5	42.7	76.7	64.2	77.6	29.0	67.8	30.7	47.9	52.0
	S-CSM	91.0	14.6	51.8	67.7	16.6	52.8	n/a	n/a	92.1	69.7	83.7	51.3	81.7	70.0	82.2	43.3	72.4	51.7	64.1	62.1
	S-BKI	93.8	24.6	60.3	76.2	21.2	65.0	n/a	n/a	92.3	73.4	84.8	54.5	83.0	71.2	83.4	47.3	73.4	56.2	67.9	66.4
Average	Sq.-KNN	87.9	14.4	43.0	61.5	57.5	32.3	55.2	63.1	92.4	59.1	80.7	43.9	77.5	61.7	76.9	34.7	71.5	32.4	47.2	57.6
	S-CSM	92.5	21.7	58.7	68.8	66.8	42.9	74.1	67.4	94.1	67.7	84.5	46.6	82.3	67.5	81.6	47.8	75.2	50.2	62.0	65.9
	S-BKI	93.4	29.2	65.8	75.4	72.9	93.0	80.7	73.7	93.7	72.3	83.8	49.0	84.1	68.7	83.0	53.7	75.8	54.0	67.1	72.1

Quantitative results on SemanticKITTI dataset sequence 00-10 for 19 semantic classes. Darknet53-kNN (Da-kNN)

Sequence	Method	Car	Bicycle	Motorcycle	Truck	Other Vehicle	Person	Bicyclist	Motorcyclist	Road	Parking	Sidewalk	Other Ground	Building	Fence	Vegetation	Trunk	Terrain	Pole	Traffic Sign	Average
00	Da-kNN	0.960	0.418	0.828	0.928	0.902	0.703	0.726	0.573	0.971	0.874	0.939	0.311	0.967	0.817	0.930	0.792	0.906	0.733	0.852	0.796
	S-CSM	0.975	0.476	0.897	0.957	0.937	0.787	0.829	0.640	0.974	0.891	0.949	0.419	0.977	0.852	0.948	0.864	0.924	0.832	0.897	0.843
	S-BKI	0.980	0.509	0.917	0.972	0.955	0.830	0.865	0.790	0.971	0.888	0.943	0.426	0.980	0.855	0.953	0.878	0.928	0.847	0.900	0.862
01	Da-kNN	0.860	0.000	0.000	0.000	0.000	0.000	0.000	0.582	0.956	0.000	0.000	0.832	0.924	0.793	0.873	0.557	0.863	0.606	0.887	0.460
	S-CSM	0.870	0.000	0.000	0.000	0.000	0.000	0.000	0.534	0.962	0.000	0.000	0.843	0.929	0.806	0.880	0.600	0.874	0.694	0.922	0.469
	S-BKI	0.889	0.000	0.000	0.000	0.000	0.000	0.000	0.586	0.963	0.000	0.000	0.849	0.959	0.825	0.895	0.691	0.883	0.747	0.942	0.486
02	Da-kNN	0.952	0.292	0.819	0.000	0.877	0.698	0.023	0.754	0.968	0.887	0.924	0.755	0.927	0.852	0.934	0.753	0.880	0.637	0.738	0.720
	S-CSM	0.961	0.339	0.859	0.000	0.908	0.798	0.020	0.754	0.966	0.894	0.924	0.780	0.942	0.861	0.943	0.824	0.895	0.723	0.807	0.747
	S-BKI	0.970	0.398	0.896	0.000	0.936	0.855	0.015	0.858	0.963	0.897	0.922	0.806	0.952	0.863	0.948	0.845	0.902	0.748	0.839	0.769
03	Da-kNN	0.943	0.393	0.000	0.712	0.889	0.473	0.000	0.000	0.971	0.828	0.935	0.000	0.930	0.880	0.951	0.645	0.922	0.722	0.782	0.630
	S-CSM	0.957	0.525	0.000	0.685	0.903	0.496	0.000	0.000	0.973	0.852	0.942	0.000	0.947	0.895	0.961	0.709	0.935	0.802	0.827	0.653
	S-BKI	0.970	0.684	0.000	0.705	0.939	0.618	0.000	0.000	0.971	0.871	0.937	0.000	0.955	0.894	0.964	0.724	0.938	0.820	0.837	0.675
04	Da-kNN	0.908	0.000	0.000	0.000	0.915	0.433	0.000	0.000	0.985	0.728	0.883	0.807	0.922	0.928	0.936	0.312	0.888	0.719	0.742	0.584
	S-CSM	0.925	0.000	0.000	0.000	0.919	0.467	0.000	0.000	0.987	0.763	0.900	0.823	0.935	0.935	0.946	0.370	0.905	0.795	0.810	0.604
	S-BKI	0.947	0.000	0.000	0.000	0.955	0.581	0.000	0.000	0.988	0.804	0.906	0.845	0.949	0.946	0.951	0.400	0.915	0.815	0.830	0.623
05	Da-kNN	0.910	0.465	0.612	0.925	0.535	0.655	0.700	0.000	0.973	0.937	0.921	0.848	0.947	0.874	0.892	0.649	0.791	0.698	0.788	0.743
	S-CSM	0.923	0.517	0.695	0.941	0.539	0.713	0.822	0.000	0.967	0.938	0.925	0.865	0.958	0.891	0.910	0.761	0.821	0.789	0.843	0.780
	S-BKI	0.938	0.591	0.774	0.957	0.563	0.820	0.913	0.000	0.974	0.944	0.928	0.876	0.972	0.900	0.926	0.798	0.842	0.807	0.869	0.810
06	Da-kNN	0.935	0.460	0.721	0.728	0.650	0.708	0.791	0.000	0.948	0.831	0.893	0.857	0.956	0.780	0.844	0.563	0.944	0.759	0.837	0.748
	S-CSM	0.955	0.547	0.810	0.742	0.675	0.838	0.923	0.000	0.963	0.851	0.922	0.884	0.971	0.833	0.876	0.711	0.959	0.880	0.931	0.804
	S-BKI	0.970	0.638	0.859	0.755	0.691	0.922	0.952	0.000	0.964	0.863	0.928	0.899	0.979	0.862	0.892	0.755	0.961	0.907	0.947	0.829
07	Da-kNN	0.961	0.446	0.876	0.911	0.939	0.768	0.798	0.000	0.970	0.888	0.946	0.000	0.971	0.816	0.894	0.769	0.910	0.750	0.869	0.762
	S-SCM	0.973	0.497	0.920	0.896	0.964	0.839	0.916	0.000	0.972	0.901	0.952	0.000	0.979	0.840	0.913	0.830	0.924	0.815	0.913	0.792
	S-BKI	0.979	0.523	0.936	0.905	0.980	0.880	0.945	0.000	0.971	0.899	0.948	0.000	0.982	0.844	0.921	0.845	0.929	0.823	0.921	0.802
08	Da-kNN	0.910	0.250	0.471	0.407	0.255	0.452	0.629	0.000	0.938	0.465	0.819	0.002	0.858	0.542	0.842	0.529	0.727	0.532	0.400	0.528
	S-CSM	0.926	0.325	0.549	0.434	0.262	0.513	0.692	0.000	0.946	0.492	0.840	0.001	0.879	0.584	0.858	0.599	0.733	0.617	0.430	0.562
	S-BKI	0.935	0.335	0.573	0.445	0.272	0.529	0.721	0.000	0.944	0.496	0.840	0.000	0.887	0.596	0.869	0.625	0.753	0.636	0.451	0.574
09	Da-kNN	0.909	0.349	0.774	0.851	0.376	0.582	0.000	0.000	0.963	0.859	0.913	0.755	0.941	0.864	0.919	0.576	0.853	0.723	0.823	0.686
	S-CSM	0.917	0.410	0.837	0.881	0.383	0.658	0.000	0.000	0.961	0.877	0.923	0.785	0.956	0.879	0.934	0.659	0.865	0.821	0.907	0.719
	S-BKI	0.932	0.490	0.887	0.904	0.392	0.723	0.000	0.000	0.962	0.875	0.925	0.797	0.966	0.897	0.943	0.678	0.888	0.848	0.920	0.738
10	Da-kNN	0.951	0.438	0.721	0.935	0.662	0.761	0.000	0.000	0.969	0.892	0.930	0.636	0.940	0.875	0.917	0.643	0.865	0.698	0.760	0.715
	S-CSM	0.965	0.466	0.801	0.950	0.679	0.841	0.000	0.000	0.974	0.909	0.944	0.670	0.959	0.898	0.935	0.743	0.890	0.796	0.841	0.751
	S-BKI	0.975	0.503	0.844	0.964	0.716	0.893	0.000	0.000	0.972	0.911	0.943	0.664	0.964	0.900	0.940	0.768	0.894	0.802	0.867	0.764

Getting Started

Building with catkin

catkin_ws/src$ git clone https://github.com/ganlumomo/BKISemanticMapping
catkin_ws/src$ cd ..
catkin_ws$ catkin_make
catkin_ws$ source ~/catkin_ws/devel/setup.bash

Building using Intel C++ compiler (optional for better speed performance)

catkin_ws$ source /opt/intel/compilers_and_libraries/linux/bin/compilervars.sh intel64
catkin_ws$ catkin_make -DCMAKE_C_COMPILER=icc -DCMAKE_CXX_COMPILER=icpc
catkin_ws$ source ~/catkin_ws/devel/setup.bash

Running the Demo

$ roslaunch semantic_bki toy_example_node.launch

Semantic Mapping using KITTI dataset

Download Data

Please download data_kitti_15 and uncompress it into the data folder.

Running

$ roslaunch semantic_bki kitti_node.launch

You will see semantic map in RViz. It also projects 3D grid onto 2D image for evaluation, stored at data/data_kitti_05/reproj_img.

Evaluation

Evaluation code is provided in kitti_evaluation.ipynb. You may modify the directory names to run it.

Semantic Mapping using SemanticKITTI dataset

Download Data

Please download semantickitti_04 and uncompress it into the data folder.

Running

$ roslaunch semantic_bki semantickitti_node.launch

You will see semantic map in RViz. It also query each ground truth point for evaluation, stored at data/semantickitti_04/evaluations.

Evaluation

Evaluation code is provided in semantickitti_evaluation.ipynb. You may modify the directory names to run it, or follow the guideline in semantic-kitti-api for evaluation.

Relevant Publications

If you found this code useful, please cite the following:

Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping (PDF)

@ARTICLE{gan2019bayesian,
author={L. {Gan} and R. {Zhang} and J. W. {Grizzle} and R. M. {Eustice} and M. {Ghaffari}},
journal={IEEE Robotics and Automation Letters},
title={Bayesian Spatial Kernel Smoothing for Scalable Dense Semantic Mapping},
year={2020},
volume={5},
number={2},
pages={790-797},
keywords={Mapping;semantic scene understanding;range sensing;RGB-D perception},
doi={10.1109/LRA.2020.2965390},
ISSN={2377-3774},
month={April},}

Learning-Aided 3-D Occupancy Mapping with Bayesian Generalized Kernel Inference (PDF)

@article{Doherty2019,
  doi = {10.1109/tro.2019.2912487},
  url = {https://doi.org/10.1109/tro.2019.2912487},
  year = {2019},
  publisher = {Institute of Electrical and Electronics Engineers ({IEEE})},
  pages = {1--14},
  author = {Kevin Doherty and Tixiao Shan and Jinkun Wang and Brendan Englot},
  title = {Learning-Aided 3-D Occupancy Mapping With Bayesian Generalized Kernel Inference},
  journal = {{IEEE} Transactions on Robotics}
}