A benchmark for monocular depth estimation in unmanned aircraft autonomous navigation, supporting obstacle avoidance and safe target tracking. The experiment videos are available here.
Our work has already been accepted by Drones and is open access at this link. Our previous work received the best poster award at ICAUS2022. The old version of source code can be found here.
The repository includes source codes of the autonomous obstacle avoidance / target tracking system proposed in our paper. Our system is specifically developed for micro-aerial vehicles (MAVs) based on monocular depth estimation (MDE), and several MDE algorithms with distinctive characteristics are applied and evaluated in our work.
It should be noted that our framework is designed with modularity in mind, allowing for seamless integration of any other MDE algorithms following ROS (Robot Operation System) development standards.
The pipeline of our system is shown as follow. Dense depth map is estimated from monocular RGB image flow firstly, while metric scale factor is calculated from UAV body height. As the depth map is recovered to metric scale, 6-DoF pose estimation and trajectory planning are implemented with ORB-SLAM2 and Fast-Planner / Fast-tracker, respectively. When flight path is generated, a path following controller is applied to track it.
Our system enables the recovery of estimated dense depth maps to metric scale. The procedure for transforming the original point cloud, generated by the estimated depth map, into a metric point cloud is illustrated in the following figure.
Six frames are included in our system, and we utilzied TF tree to manage all the coordinates. The definition relationships between frames are shown as follow.
We applied and evaluated several MDE algorithms in our work. The estimated depth map was transform into point cloud by camera intrinsics, and the ground was segmented by RANSAC algorithm. Distances between the UAV and detected obdtacles are also recorded to evaluate each algorithm estimation performance.
According to our previous experimental results, we figured it out that both depth accuracy and scale consistency play significant roles in real-world MAV navigation, and SC-DepthV2 performed best in the three selected algorithms due to its scale-consistency.
We have tested all the packages in ROS Melodic on Ubuntu 18.04. We recommend to manage the MDE workspaces with Anaconda Virtual Enviroment.
Our system supportes tasks like autonomous obstacle avoidance and target tracking. Following content presents the running workflow of our system.
General Setup introduces fundamental packages used in both autonomous obstacle avoidance and target tracking. Tello Driver (rmtt_driver_ws) is a package should be launched firstly to receive monocular RGB image flow and UAV body metric height. Monocular Depth Estimation (monocular_depth_ros) comprises multiple integrated MDE algorithms (MonoDepth, MiDaS, and SC-DepthV2), each requiring specific configurations for operation. Scale Recovery (scale_recovery_ws) offers task-specific functions based on the chosen task (MDE Performance Evaluation, Autonomous Obstacle Avoidance, or Target Tracking). After that, UAV Pose Estimation (orb_slam_2_ws) can be accomplished.
According to the application requirements, please select either Autonomous Obstacle Avoidance or Target Tracking. The main distinction between these two options lies in how the goal is determined - manually specified in Autonomous Obstacle Avoidance, and obtained through target detection in Target Tracking.
Autonomous Obstacle Avoidance applies Fast-Planner (fast_planner_ws) to generate a safe flight trajectory to avoid obstacles when a manual goal is set for the UAV to reach.
Target Tracking obtains goals from Target Pose Estimation (target_detection_ws) constantly, and leverages Fast-tracker (fast_tracker_ws) to plan flight trajectorys simutaneously, which allows the system to both avoid collisions and maintain tracking of the target even in cases where it may be temporarily lost.
Velocity Contoller (mav_controller_ws) generates velocity commands to control the MAV.
NOTE: Prior to running, ensure that each package is compiled. For more detailed information regarding setup, compilation, and operation, refer to the corresponding readme file within each workspace folder.
A Tello driver should be launched at first. After compilation, refer to rmtt_driver_ws, please run:
cd ~/Mono_Drone_Eva/rmtt_driver_ws
conda activate py38
source devel_isolated/setup.bash
roslaunch rmtt_driver rmtt_bringup.launch
To get the UAV off the ground and control it via keyboards, please open a new terminal and run:
cd ~/Mono_Drone_Eva/rmtt_driver_ws
conda activate py38
source devel_isolated/setup.bash
roslaunch rmtt_teleop rmtt_teleop_key.launch
Then follow the tutorial of rmtt_teleop to control the UAV.
Estimated dense depth maps will obtained in these packages. Due to specific configuration requierments, please refer to monocular_depth_ros to complete operation.
This package calculates metric scale factor from estimated depth map and metric height obtained by ToF sensor. Take target tracking as an example, after compilation, please run :
cd ~/Mono_Drone_Eva/scale_recovery_ws
source devel/setup.bash
roslaunch scale_recovery scale_recovery_tracker.launch
Some other functions (such as MDE performance evaluation) are also provided in this package. For more details, please refer to scale_recovery_ws.
ORB-SLAM2 is applied to estimate UAV pose. After compilation, refer to orb_slam_2_ws, please run:
cd ~/Mono_Drone_Eva/orb_slam_2_ws
source devel_isolated/setup.bash
roslaunch orb_slam2_ros orb_slam2_rgbd.launch
After compilation, you can initiate the visualization and run the program directly by:
cd ~/Mono_Drone_Eva/fast_planner_ws
source devel/setup.bash
roslaunch plan_manage kino_replan_tello.launch
See fast_planner_ws for more detailed information.
Please run the following commands to execute AprilTag pose estimation :
cd ~/Mono_Drone_Eva/target_detection_ws
source devel_isolated/setup.bash
roslaunch rmtt_apriltag detection.launch
NOTE: THIS NODE SHOULD BE LAUNCHED AT THE FINAL STEP.
Seetarget_detection_ws for more details.
Similar to Fast Planner, run
cd ~/Mono_Drone_Eva/fast_tracker_ws
source devel/setup.bash
roslaunch plan_manage tracking_model_tello.launch
to visualize grid map and start tracking. Please check fast_tracker_ws for detailed information.
Two controller are included in mav_controller_ws: a PID controller and a path following controller.
The PID controller is recommended to use in autonomous obstacle avoidance with Fast-Planner. For pid_controller, please run :
cd ~/Mono_Drone_Eva/mav_controller_ws
source devel/setup.bash
roslaunch mav_controller pid_controller.launch
The path following controller is a appropriate choice for target tracking with Fast-tracker. To run the path following controller, please type :
cd ~/Mono_Drone_Eva/mav_controller_ws
source devel/setup.bash
roslaunch mav_controller stanley_controller.launch
If you publish work based on, or using, this code, we would appreciate the citation to the following:
@article{pan2024mona,
title={MoNA Bench: A Benchmark for Monocular Depth Estimation in Navigation of Autonomous Unmanned Aircraft System},
author={Pan, Yongzhou and Liu, Binhong and Liu, Zhen and Shen, Hao and Xu, Jianyu and Fu, Wenxing and Yang, Tao},
journal={Drones},
volume={8},
number={2},
pages={66},
year={2024},
publisher={Multidisciplinary Digital Publishing Institute}
}
@inproceedings{pan2022does,
title={How Does Monocular Depth Estimation Work for MAV Navigation in the Real World?},
author={Pan, Yongzhou and Wang, Jingjing and Chen, Fengnan and Lin, Zheng and Zhang, Siyao and Yang, Tao},
booktitle={International Conference on Autonomous Unmanned Systems},
pages={3763--3771},
year={2022},
organization={Springer}
}
For any technical problems, please raise issues or contact Yongzhou Pan ([email protected]).
The test methods come with their own license, and the rest follow the GPLv3 license.