Look Gauss, No Pose! - Gaussian Splatting without accurate pose information

This repository contains code for the IROS24 paper "Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization" by Christian Schmidt, Jens Piekenbrinck, and Bastian Leibe. Arxiv

Installation

Clone the repository with submodules: git clone https://github.com/Schmiddo/NoPoseGS.git --recursive. The renderer needs a working cuda compiler; it should work with all versions 11.8+. Run the following commands to install the needed requirements:

pip install -r requirements.txt
pip install submodules/lietorch submodules/cudaops submodules/simple-knn submodules/diff-gaussian-rasterization --no-build-isolation

Compiling all extensions might take several minutes. If you utilize multiple different GPUs, we recommend to add all needed compute capabilities to the TORCH_CUDA_ARCH_LIST environment variable before installing the extensions, e.g., add export TORCH_CUDA_ARCH_LIST="6.1;7.5;8.6;8.9" to support 1080, TitanRTX, 3090, and 4090 GPUs.

Data preparation

For the experiments, we use three datasets: LLFF, Replica, and Tanks&Temples. You will need to download the data and precompute monocular depth estimates for all frames.

Data download

• The LLFF dataset was originally linked in the LLFF repo, but the link seems to be broken at the moment. Alternatively, you can download it from kaggle. Our dataloading scripts expects the down-scaled images to have the same names as the original ones; you can run for d in <path-to-llff>/* ; do python rename.py $d ; done to create hardlinks with the correct name.
• For replica, we use data generated by iMap and hosted by Nice-Slam; download here.
• For Tanks&Temples, we use the data provided by NopeNerf.

Extract all of these datasets to data/ or point the environment variable $DATA_DIR to your data directory.

Compute depth estimates

We use the same depth estimator as NopeNerf, from "Vision Transformers for Dense Prediction" (DPT). First download the weights from here and put them under models/ or point $MODELDIR to the directory containing the weights. Then run python -m scripts.generate_depth -llff -rep -tatnn to precompute depth estimates.

Training/Evaluation

We provide two experiment setups: camera pose estimation with respect to a trained model, and optimizing a scene without accurate camera poses. To reproduce the first set of experiments, run ./run/noposegs_camopt.sh. This script creates models for all scenes of the llff dataset and then runs the pose estimation experiments. To run the second set of experiments, run ./run/noposegs.sh. This script runs joint reconstruction and camera pose refinement on llff, replica, and tanks & temples. We also provide run scripts for the baseline, vanilla 3DGS from COLMAP poses (optionally with noise), using ./run/3dgs.sh.

🎓 Citation

If you find our work useful, please cite our paper:

@inproceedings{schmidt2024noposegs,
author     = {Schmidt, Christian and Piekenbrinck, Jens and Leibe, Bastian},
title      = {Look Gauss, No Pose: Novel View Synthesis using Gaussian Splatting without Accurate Pose Initialization},
booktitle  = {IROS},
year       = {2024},
}

References

This repository is based on the original 3DGS implementation:

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, George Drettakis: 3D Gaussian Splatting for Real-Time Radiance Field Rendering (SIGGRAPH 2023)