NexusSplats: Efficient 3D Gaussian Splatting in the Wild

Yuzhou Tang · Dejun Xu · Yongjie Hou · Zhenzhong Wang · Min Jiang*

Xiamen University

News

[2025.2.20] 😊 We have supported custom datasets with COLMAP format in NexusSplats.

[2025.1.16] 🤗 We have released the pretrained models for NexusSplats in #release.

[2024.11.25] 🎈 We have released the codebase for NexusSplats.

TODO List

• Codebase release
• Pretrained models
• Merge with the latest version of NerfBaselines

Abstract

Photorealistic 3D reconstruction of unstructured real-world scenes remains challenging due to complex illumination variations and transient occlusions. Existing methods based on Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) struggle with inefficient light decoupling and structure-agnostic occlusion handling. To address these limitations, we propose NexusSplats, an approach tailored for efficient and high-fidelity 3D scene reconstruction under complex lighting and occlusion conditions. In particular, NexusSplats leverages a hierarchical light decoupling strategy that performs centralized appearance learning, efficiently and effectively decoupling varying lighting conditions. Furthermore, a structure-aware occlusion handling mechanism is developed, establishing a nexus between 3D and 2D structures for fine-grained occlusion handling. Experimental results demonstrate that NexusSplats achieves state-of-the-art rendering quality and reduces the number of total parameters by 65.4%, leading to 2.7× faster reconstruction.

Overview of NexusSplats. Our framework operates in three stages: First, the Hierarchical Gaussian Management organizes 3D Gaussians into dynamic nexus kernels, which generate Gaussian attributes and perform Centralized Appearance Learning and Uncertainty Propagation. Second, a raw image, a mapped image, and an uncertainty mask are rendered though tile rasterization. Third, the Boundary-Aware Refinement corrects misclassified scene boundaries. The system optimizes via a combination of color loss and uncertainty loss.

Installation

Clone the repository and create a python == 3.11 Anaconda environment with CUDA toolkit 11.8. Our code is implemented based on NerfBaselines. Install the dependencies and the codebase:

git clone [email protected]:juliantang324/NexusSplats.git
cd NexusSplats

conda create -y -n ns python=3.11
conda activate ns
conda env config vars set NERFBASELINES_BACKEND=python
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url https://download.pytorch.org/whl/cu118
pip install --upgrade pip
pip install -r requirements.txt
pip install nerfbaselines==1.2.5
pip install -e ./submodules/diff-gaussian-rasterization ./submodules/simple-knn
pip install -e .

Training

To start the training on the Photo Tourism dataset, run one of following commands:

ns train --data external://phototourism/trevi-fountain
ns train --data external://phototourism/sacre-coeur
ns train --data external://phototourism/brandenburg-gate

Evaluation

To evaluate the trained model on the Photo Tourism dataset, run the following commands:

# render predictions
ns render --checkpoint {checkpoint} --data external://phototourism/trevi-fountain --output {output_path}
ns render --checkpoint {checkpoint} --data external://phototourism/sacre-coeur --output {output_path}
ns render --checkpoint {checkpoint} --data external://phototourism/brandenburg-gate --output {output_path}

# evaluate predictions
ns evaluate {path/to/predictions} --output results.json

Interactive Viewer

To start the viewer and explore the trained models, run one of the following commands:

ns viewer --checkpoint {checkpoint} --data external://phototourism/trevi-fountain
ns viewer --checkpoint {checkpoint} --data external://phototourism/sacre-coeur
ns viewer --checkpoint {checkpoint} --data external://phototourism/brandenburg-gate

Concurrent works

There are several concurrent works that also aim to extend 3DGS to handle in-the-wild scenarios:

• WildGaussians: 3D Gaussian Splatting in the Wild
• Gaussian in the Wild: 3D Gaussian Splatting for Unconstrained Image Collections

Acknowledgements

We sincerely appreciate the authors of 3DGS and NerfBaselines for their great work and released code. Please follow their licenses when using our code.

Citation

If you find our code or paper useful, please star this repository and cite:

@article{tang2024nexussplats,
    title={NexusSplats: Efficient 3D Gaussian Splatting in the Wild},
    author={Tang, Yuzhou and Xu, Dejun and Hou, Yongjie and Wang, Zhenzhong and Jiang, Min},
    journal={arXiv preprint arXiv:2411.14514},
    year={2024}
}