Seamless Augmented Reality Integration in Arthroscopy: A Pipeline for Articular Reconstruction and Guidance
Arthroscopy is a minimally invasive surgical procedure used to diagnose and treat joint problems. The clinical workflow of arthroscopy typically involves inserting an arthroscope into the joint through a small incision, during which surgeons navigate and operate largely by relying on their visual assessment through the arthroscope. However, the arthroscope's restricted field of view and lack of depth perception pose challenges in navigating complex articular structures and achieving surgical precision during procedures. Aiming at enhancing intraoperative awareness, we present a robust pipeline that incorporates simultaneous localization and mapping, depth estimation, and 3D Gaussian splatting to realistically reconstruct intra-articular structures solely based on monocular arthroscope video. Extending 3D reconstruction to Augmented Reality (AR) applications, our solution offers AR assistance for articular notch measurement and annotation anchoring in a human-in-the-loop manner. Compared to traditional Structure-from-Motion and Neural Radiance Field-based methods, our pipeline achieves dense 3D reconstruction and competitive rendering fidelity with explicit 3D representation in 7 minutes on average. When evaluated on four phantom datasets, our method achieves RMSE = 2.21mm reconstruction error, PSNR = 32.86 and SSIM = 0.89 on average. Because our pipeline enables AR reconstruction and guidance directly from monocular arthroscopy without any additional data and/or hardware, our solution may hold the potential for enhancing intraoperative awareness and facilitating surgical precision in arthroscopy. Our AR measurement tool achieves accuracy within 1.59 +/- 1.81mm and the AR annotation tool achieves a mIoU of 0.721.
关节镜是一种用于诊断和治疗关节问题的微创手术。关节镜的临床流程通常涉及通过小切口将关节镜插入关节，手术过程中外科医生主要依赖关节镜提供的视觉评估来进行导航和操作。然而，关节镜的视野受限及缺乏深度感知，在处理复杂的关节结构时增加了导航难度，并影响手术精度。为了增强术中认知，我们提出了一个鲁棒的工作流程，结合了同时定位与建图（SLAM）、深度估计以及3D高斯分布（3D Gaussian Splatting），通过单目关节镜视频真实地重建关节内结构。将3D重建扩展到增强现实（AR）应用中，我们的解决方案提供了AR辅助功能，用于关节切迹测量和标注锚定，并支持人机协作的操作模式。与传统的基于运动结构（Structure-from-Motion）和神经辐射场（Neural Radiance Field）方法相比，我们的流程在平均7分钟内实现了密集的3D重建和具有竞争力的渲染保真度，提供明确的3D表示。在四个仿真数据集上进行评估时，我们的方法平均重建误差为RMSE = 2.21mm，峰值信噪比（PSNR）为32.86，结构相似性（SSIM）为0.89。由于我们的流程能够直接从单目关节镜视频实现AR重建和指导，无需额外的数据或硬件，因此我们的解决方案可能有助于增强术中认知并提高关节镜手术的精度。我们的AR测量工具的精度为1.59 +/- 1.81mm，AR标注工具的平均交并比（mIoU）为0.721。